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Software Research Associates (5A). 
Results indicate that the gr2atesce differences between 
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group's interface with the software design and development 
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ie LNT RO DUGE TON 


Be, CHE PROBLEM 


Software quality assurance is the planned and systematic 
Bee sons cegquired to proviie adequat: confiderzs that so 


Were products conform ¢> standaris established by «he 


company developing the proijuct and the contractual require- 
BopeeceepsOV2dSd by he customer [RSF. 1}. This phenomenon 
crosses all customer bouidgaries: Sor tenes cee Tas: = eae 


military, other governments; and crosses different applica- 
2ion types: operating systems, infornation systems, process 
Sencrol, Command sand coyttol, SS AMumeca= 2 on, business 
systems, etc [Ref. 2]. 

In the United States Navy (USN), the office in charge of 
design, development and life cycle] maintenance 2s the supply 
System computer network is the Fleat Material Sipport Office 
(FMSO) in Mechanicsburg, Pannsylvani:3. Oreos Oczob eri OCI 
MoO Ss Coumending Office: sstablishei a gualizy program *ask 
Mem@ewewho ch Consistsd of AutoOmatis Data Processing (ADP) 
technical personnel fron 32a 
(CDA) d¢partments and supoorting Jd229artmencs. L=s  wwspos2 
Was to consider quality in a broad sense as it rela 
the ADP system development process and <9 outline 2a asneral 
Cie ectea Yeableanmd cont@nuling gualizy progren. iE 
@eluo'S Main objectives were +o provid> rtecommendagiors that 

© 


WeomeegmemorOys the gqualicy of FASO products, accdunt f9r this 
u 


qiwemeyeoeOcess and €ustain it thEoughout the product's iifs 
meee ene Conclusions 3£ the task jrsup were: 
leer Olina 3)* improvement was p2ssible oe eee FMSO 


=eyo es onmen=. 





Peeeeiiee-y accountability was raguired and was becoming 
o 


a 

increasingly important. Correstly perforned, mneasure- 

Boece eOuld ftesmai= in an sffactive ana ccountableé 
quality process. 

3. The ability to sustain acceptable lavels of quality in 

an environment of changing technology can be accommo- 

Gaeesae~- nbOugh the 2ferative accounting ani analysis of 

Emoauc-avVity end inventory characteristics. — Ref. _ 

Dillteeng th2S same fame period, twd other special project 
were receiving major attention by FMSO. ore is «ha 
Reso u= citation DEG wecumes 1 UGh ijjiantifies thes computer 


Beq@emem=nts at the two Tivéentory Control Points (IIP's) in 


moe togO'S and beyond, tiking ints account both «he near 
Saturetion of the present JInivac 494 somputers at the ICP's 
aed their Obsolescence. The othr Paod-c-, called 


tRasystemization," is also a massiv= undertaking as it will 
eventually reasul= in new software or somputer proatams for 
*he ICP's. Talks between the author and FMSO's Commanding 
Oretecermematcate that this projec: “Ref. 4} gives FMSO more 
Migeomemve tO “ake a serisus 159k a& its quality assurance 


program. 


pee ee OS & 


The purpose of this thesis is to investigqat? the methods 
Sco elaine commercial computer companies i the area of 


Ss in € 
Comendremqualicy 2aSsStivancs.™” The prinary objective is *o se2 
eee of these "practices can = US 4 


environment. 


9, 





C. METHODOLOGY 


Mice pEOcsaqure used £5 accomplish the thesis objeckive 
was to interview personnel from the various conputer compa- 
miss. The following compaiies' personnel were interviewed: 

1. International Business Machines (IBM) Corporation, 

2. TRW Incorporated, 
3. Hewlet+ Packard Company, 
fe Mamdehl Corporation, 
5. Software Research Associates (SRA). 
These companies were chosen because they are located near 
the Naval Postgraduate School in Monterey, CA and they gives 
a broad view of the conputer software industry. The 
following questions were asked at tha interview: 
1. Where does the guality assuraisS= group fit into ths 
OEgan i Zation? 
Peeeeana. <ype Of alithcrity/pow2ar ise¢s the guality assur- 
ance group have over the software pro 
Bea nat qualifications do th 9s9ple in the quality 
assurance group have? 
4%. How does the gualit assurance? group interrace with 
the design/developm2at group? 
Seeea- =O0lS, methodologies, or techniques ioes <zhe 

G@uciity assurance group use t59 d> “=h 5 
Semeace historical records kept of problems with softwares 

products after theic release and who in the company's 

Organization keeps them? 

Teeanho handles problems with software after release, and 
how are such problens handlsi? 
Seeemo me DEand new oroduct 2s design2d, who in the compa- 

Payson Geni zation =fains 2h customer on this product? 


The data was then comparsi with existing praccricss at FMSO 
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tee seaucCTURS OF THE THESIS 


Chapter I, the introduction *5 the thesis, presents the 
thesis objective and mathodology. Shapter II presents 3 
general literature review of the problem of quality assur- 
ance and the factors that are taken into consiizéeration when 
Gefining it. Chapter III addrasses tha FMSO environment and 
ime Deecess of quality control. SThapter IV presents the 
interviews conducted with the personiel of the five computer 
organizations as to their software guality assuranc=: 
zations and how they work. The final chapter offers 23 
Summary of «nese interviews and provides recomnendations on 


how these ideas miaqht be aoplied at FMSO. 
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II. SURVEY OF LITERATURE 
Chapter It deals with the problem of softwar2 quality 
assurance. Rie eeera Com DuTe— S2 asc sO mM ohind \veueren. 
literature on this subject, it was discovared zha+ all 


Seder o Of these writings failed £5 agree on the definition 


of pertinent terms. In order to jefine the terms relevant 
mo ‘this thesis, the author oprasents fhe <tolloging 


@eranst2ons . 


A. DEFINITIONS 


SOmemare 35 a sét 3£ coded instructions which are 
supplied to and operate with the tonputer hardware to cause 
the hardware to perforn the functions defined in the 
Miserider rons. (Ref. 5) 

A system, as definei by ths Pleet Material Support 
Office, is an organized set ofr Automatic Data Processing 
(ADP) hardware, environnental and apolication software, and 


dccumented procedures designed to 2utomate th2 basis manage- 
= 


ment and »perating processes for a customer sit? or group of 


customer Sites with common mission responsibilities 
fRef. 6]. "Documented prosedures,” 25 used above, refers +o 


th2 applicable ADP-relat2i and non-ADP-related procedures 
established to support <*h2 hardware and software aspects of 
*he system, 2.g. “he computer speration manual and «he users 
Manual { Ref. 7}. 

Quality assurance of hardware has been su 
accomplished for many years, but there ars major differences 
between hardware and software: 

1. Scftware develcpmant specifications are usually nocz as 
Or a S 


Seecetic as thos2 £ 


a 





terms wit unspecified Jefinitions such as "optimum" 
or "99.9 percent reliable" ar> used which are poten- 
it u 


2£5 Ofee tne software 


Ya 
(D 
2 
VY 
J> 
O 
a! 
O 
ty 


tial seeds of dis 2WS 
1s produced. 

Mme Crt were product (bailil+-to) specifications are usually 
less rigorous. 

3. The softwars development procss= is also the produc- 
tion process because there are n> bread boards, brass 
Beards, DhOLGtTyYpes Or pre-production models t5 use. 

Seer ne production of ssfEtware (coie2} is neither a fully 
censtrained nor a uniquely defined process. 

5. The software product itself (r29d@) is essentially an 
Mmeanel ple substance With forn, conten*®, and functions 
manifested via images. 

Seamcert Ware problem fixes alwayse tesult in a product 
Sonelguration Change. Ref. 8] 

A basic software development prossss is shown in Figure 
Di lie Corporat? analyses 39 life-cycle ccest have shown that 
the cost of maintenance and redesign exceed the cost of 
mie d2=vyelopment and shat the cost of fixing error 
the software is operational is up ts 30 times grea 


fe inet ernacstogeal «'s 


h 

MOE rreSo ting errons ducing systen *2s*ing. FaCWce G2. 
shows a summary of experience u e 
a 


Machines, General Telecomnunic Equipment (GIrE), 


8) 
ct 
+ 
Q Oo ww 
a | 
YI 


Meaamor ops selatiye cost »9f correcting software errors as 3 
mumettom OL the phase in w#hach they 2252 corrected. Figure 
Meeesiiages<=S that =f oOayYS <9 21 
SSa-ching fort errors during «the earl 
wee wousp=pa T00-nan holzgs ,. corres 


svyorem 2S in Operation. [Ret. 9] 
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TEST TEST 


PHaSE IN WHICH ERROR DETECTED 


FIGURE 2.2 The Price of Procrastination in Error Detection 


SOURCE: Dr. Barry W. Boehm's Article on Software Engineering, 
1 June 1981 
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DemOUALITY FACTORS AND CRITERIA 


~ecemurE Ge tactOls Gon-cributs <6 ths gquality of software. 
Eleven of these factors are defined i Figuce 2.3 The 


rationale (Ref. 10} behini the choice of these is one of 
Metiity; Cachn facton identified could be applied to 3 
production environmen. PReeIncerastlonweOL support e) 
Within an operational environment involves three di 
de= vei= les: DOMUGtT "Operation, produce revision, and 
eaoddet transition. Pigure 2.4 shows a conceptual schens 
With these three activiti2s and some related questions which 
snvolve the quality factors (Ref. 11). 

imese quality  factozs can be farther broken io 
Criteria which could be usad for oth2r purposes. 2 
Seewommc=.teria for each factor further defines ths 


Second, the criteria which affect more “han ons factor help 
h 


describe the relationships between the factors, and the 
criteria establish a working hierarchical framewerk for 
factors in software quality. Thesa criteria ac2 defined and 


their relations «9 factors are shown in Piaqur 

DPescimyeee With the use |Ref. 12] of these factors and thsir 
criteria, 2 possible nunerical value mav be added to help 
forecast the quality of tha softwar during its develcpment 
Syeloueernis is the goal o£ softwares wetrics, aweool used by 


sone companies for this parpese (Ref. 13]. 


SeeeosreeaAl RESPONSTIBILITESS AND ORGANIZATION 


@eompenicses are finding that it is advantagesus, from both 
ic 


t 
pageiwecequality and cost-s£ 


ectiveness standpoints, <9 haves 

foe, EXP LIci~ gqualit ASSeete- saccuysty One ineir SScEwacs 

projects [ Ref. 14]. Die se ccka Orgies aGeiVicy ate usually 

mampeeeeec ©O <hS project aid depend 21 size and scope. iiss 
ve J 


Bepecgdeh has vroved effertiv 
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CORRECTNESS 


RELIABILITY 


YP CM Ty ex 


INTEGRITY 


USABILITY 


MAINTAINABILITY 


TESTABILATY 


SLeIeILITY 


rORTABILITY 


PEUSABILITY 
a 


a e Ler rABLiua. pe 


Extent to which a program satisfies its specifications 

and fulfills the user's missicn objectives. 

Extent to which a program can be expected to perforn its 
intended function with required precision. 

The amount of computing resources and code required Ly 

a program to perform a functicn. 

Extent to which access to software or data by unauthorized 
cersons can be controlled. 

Effort required to learn, operate, prepare input, and 
interpret output of a program. 

Effort required to locate and fix an error in an orerational 
program. 

Effort required to test a program to insure it perrorms 

its intended ftumcticn. 

Effort required co modify an cperaticnal orocram. 

Effort required to transfer a program <rom one harcware 
configuration and/or scftware system environment to ancther. 
Extent to which a crogram can be used in cther applications 
related to the packaging and scope of the fimctions chat 
orograms Serfomn. 


Sffore required to courle cne system with another. 


FIGURE 2.3 Definition of Software Quality Factors 
SOURCE: Macabe's Book on Software Quality Assurance - A Survey 
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RELIABILITY 


Consistency Accuracy 


Traceability 
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Execution Efficiency Storage efficiency 





LEGEND 


cD Factor 
C= Criteria 
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FIGURE 2.5 Relationship of Criteria to Software Quality 
Factors 


SOURCE: Macabe's Book on Software Quality Assurance - A 
Survey 
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FIGURE 2.5 Relationship of Criteria to S I 
oftwa 
Factors (Contd.) Sites 


SOURCE: Macabe's Book on Software Quality Assurance - A 
Survey - 





: RELATED 
CRITERION OEFINITION FACTORS 


TRACEABILITY Those attributes of the software that provide! Correctness 
a thread from the requirements to the imple- 
mentation with respect to the specific 
development and operational environment. 


COMPLETENESS Those attributes of the software that Correctness 
provide full implementation of the functions 
required. 

CONSISTENCY Those attributes of the software that Correctness 
provide uniform design and implementation Reliability 
techniques and notation. Maintatnability 


ACCURACY Those attributes of the software that Reliability 
provide the required precision in calcula~ 


tions and outputs. 















ERROR TOLERANCE | Those attributes of the software that Reliability 
provide continuity of operation under 
nonnominal conditions. 

SIMPLICITY Those attributes of the software that Reliability 
provide imolementation of functions in the Maintainability 
most understandable manner. (Usually Testability 


avoidance of practices which increase 
complexity.) 





MODULARITY Those attributes of the software that Maintainability 
provide a structure of highly independent Flexibility 
modules. Testability 


Porta) ity | 
Reusabdility | 


| 


Intercoerability | 






Flexibility 


Reusabi lity | 


GENERALITY Those attributes of the software that 
provide breadth to the functions performed. 


provide for expansion of data storage 

requirements or computational functions. Fiexibiiity 
INSTRUMENTATION | Those attributes of the software that Testability | 
provide for the measurement of usage or 


identification of errors. 


Those attributes of the scftware that 
provide explanation of the imolementation 
of 4 ,anesion., 


Piextov lity 

Matias 1a. tev 
eeStaoll 27 
chen eer toys) Malek 
Yeusaorlity 


e 
Serer VENESS 





a ee ee 


EXPANDABILITY Those attributes of the software that | 





rod 


FIGURE 2.6 Criteria Definitions for Software Quality Factors 


SOURCE: Macabe's Book on Software Quality Assurance - A 
Survey 
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‘ | on RELATED 
CRITERION DEFINITION | ell 






2 XECU7 TON 
Serle. eicy 


Those attributes orf the software tnat 


Efficiency 
provide for minimum Jrocessing time. 





























STORAGE Those attributes of the software that cfficiency | 
ape eNCY provide for minimum storage requirements 
during operation. 
A@ecss CONTROL | Those attributes of the software that | Integrity 
provide for control of the access of 
software and data. 
aCCcSS AUOIT Those attributes of the software that Integrity 
orovide for an audit of the access of 
software and data. 
CPE2ABILITY Those attributes of the software that Usability 
determine operation and procedures con- 
cerned with the operation of the software. | 
TRAINING Those attributes of the software that ws) | 1ty 
Orovide transition from current operation 
| or initial familiarization. 
| COMMUNICATIVENESS Those attributes of the software that Usability 
| provide useful inputs and outputs wnich 
can be assimilated. | 
| SSI T AIS, Sy esi! Those attributes of tne software that Morcabllicy 
INDEPENDENCE determine its dependency on the software Reusabiiity | 
environment (operating systems, utilities, 
input/output routines, etc.) 
| MACHINE those attributes of the software tnat | Poreaol lity 
Peobere IDE CS determine its denendency on the nardware | Reusadilicty | 
| systen. | 
ee eee ee EE ee ee ee 
COMMUNICATIONS Those attributes cf the software that Interoperacility . 
ComOtlal ctf provide the use of standard protocols 
ands ancver-ace routines. ; 
| SATA COMMOHAL TY Those attributas of tne sortware that iN<erOserac? acy 
| orovide the use of standard data reore- 
sentsations. 
power ceucsS | Those attrébutes of tre software cat | “aintainadiiity | 
| arevide «ome itohememcacionec. 2 "2rSr ion | | 
| Alon ao ming aun amour ecce. | 
i a naan 


FIGURE 2.6 Criteria Definitions for Software Quality Factors 
(Contd. ) 


SOURCE: Macabe's Book on Software Quality Assurance - A Survey 
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2S responsive to the quality requicements of the customer 


and 


Pome meomePaecleuler system application: The general 


Beepomcabitaties of Such an activity include: 


ile 


ON 
e 


Planning 

PemereGepares20n Of Ehe quality assurance plan stasing 
duties, responsibilities, ani schedule. 

b) Project and custoner interfaces. 

c) Resource management. 

d) Subcontractor/supplier manayement. 

Policy, Practice 2nd Procedure? Dev 

a) Preparation of standards manual for all phases of 
“=a eesOoLtWwale production, including requiren 
design, EOGIUG, mend ©2St tailored” *2o- speciric 
project requirements. 

b) Problem reporting and analyses. 

Seftware Quality Assurance Aids Develonmeant 

EMeantatloheo: 2sxisting tosis or metheds. 

b) Development of manual and automated procedures. 

c) Keeping abreast of new and "State of tha art" aids. 

Audits 

a) Review cf project procedur2s and decumantatior for 
eompinance tos 2a dards. 

by Participation in interim teviews. 

Speer accleipation in customer audits of the preject. 

d) Quality assurance inspections. 

Test Surveillance 

erect iG. patton in Ehe testing phase. 

Bien ehe2=inhg of soOttva 


S§sod aeSevsanrc=. Of -CO== 2c- 


ry 
Q 


GyeeanelySis 2f E279 aus 
Vereact 102s 
Records Retention 


euemeomacla2=y asstutancs records ninagement. 


2u 





Dy Retention of problem reports, *est cases, t2st 
BoeeweeeOgs Sf quality assurance rev 

See eolr> Proper documentation. 

Physical Média Control 


~~] 
e 


a) Inspection of jiisk, tapes, cards, and Weane: 
program-retaining media - verification at all times 
SL epiysical tramsmettal or rstention. 

Die CLOtection ig mishandling of altering by 
environment. [{R2af. 15] 

The classical quality assurance gro 
with the development cycl2 usually so 
development cycle when testing stacts. li) ok Gea (@. 
dissect the problem, fini errors, test for the environnent 
euewoach the software product is t> be used in and notify 
the developers of fauits. This sometimes produces an adver- 
Sary relationship between th? groups, isstroyiag any cocper- 
ation or aid one might give the othéc. The autonomy of “he 
quality assurance group is aiso imperative for achieving any 
“ype of success. [{Ref. 15] 

Meme@neware Production environments *oday, th gquelity 
assurance group's intention is to work with tha development 
meemor the house throughout the dev 
view themselves as a tool or aid ¢t> *h2 managemen-= of the 


levesworpmer= process, informing the aanager of problems «hsv 


Be-meeaS @ hinderance to the schedule or quality of ths 
product under developmen. THegoueoRemy Cr fits) *gsO0un. is 


Bes eimportzant. {f{Ref. 17] 


D. SUMMARY 


jMimemechapres has listed the questiors Which must 09 
pede ted abou+ the softwares product before the duties of th 
Meme y assusence group cain be deliniatsd. A 

eee ene sxact roles 9£ =h2 quality assurance group 42ni 


oo ww ~~ 


bo 
e7) 





its interfaces with the levelopment group may be viewed 
differentiy, depending on tha character of the organization 


itself. The following chapters define the purpose and envi- 


ronments of the quality aSSuranz2 organizations under 
consideration and explain theirc process of quality 
assurance. 


Zo 





III. FLEET MATERIAL SIPPIRT OFFICE 

Riemer uspOoss of this shapter is to 

environment and the process of softwa 

ees eomgen® Zazion. The following refere 
fee fet corzal Suppsftt Offices Irganizational anual 

eee et Maverial Supodr+ Jffiss Centsal Design Agency 

(CDA) Management Handbook, 1 January 1981, 

Seeeeceo. Material Susp0rt Office Internal Instruction 

9239.20A CDA Daveloonent Hand 1 Decsmber 1979, 


Weer necet Material Support Offic= Internal Instruction 


oO 
UU 
O 
~~ 

= 


9230.12 Quality AsSSicance Program, 17 May 1978, 

te fleet Material Support Offices Quality Program Task 
GE@up Report, 31 January 1982, 

9. Ine Navy Supply farps Newsletter e2avery i762, 


Special Issue "Celebrating FYS)'s 20*h Anaiversary." 


per to LORY 


Established in 1962, FMS) was originally chartered to 


Peovade céntzai management for th2 r2ztail portion of «the 


Meyy Secck Fund (NSF). i: was also ised to obtain and stock 
Supplies from other services. femea tt Soucerelovlca, Gace sor 
Supply Ssysten performaace analysis and evaluation. 


5 


Meregenem ly =his »°yanization consisted of five »sfficsrs and 
femme=Vilten emplovees, bit <tcday fe) 


mtieeeieery personnel and »vez 1,309 civilians. 
Prema ns tTeasOkR fOr the Coganizition's growth has beer 


Mesmeemerease in responsibilities. Thee = 554.26 


d 
6ccured in 1965 when the TZentral Design Agency Junction was 
meet pOlra-~ed into its It volyv 

Cc 


ea=)) G6S9n, develop 





programs used in computer systems. This initial designation 

was limited to computer systems us2?4 in supply and financial 

op2rations at various field? a 
eee o7 3s, ° £MSO's direct r 

increased with the assignment 

+ion was reassigned to th2 Nav of 

a7 1978. In 1977, twd additzonal increases in FMNSoO!t 

mission area occurred. The “Bc naaectal «sy 

Significantly expanded with ths assignment o 


S 
A O 

bility for financial sys<2as utilizei by headquarters activ- 
i: 


meses in WaShington, Oye. Such, 4S <the-Nava Material 
Command and various systens commands. The other 2xoansion 
Magewene fcesult of FMSO's designation as the CDA for the 


Trident Logistics Data System, which added submarine inter- 
mediate level maintenance to FMSO's CDA mission. The most 
meeemc 2adai tion £5 their nission actea occurred in 1978 with 
*h2 responsibility assignment of che prototyp? development 
=Or the Naval Aviatioa Fog Ss * 2c se connard Management 
ienomrma.20n System {NALCOWTS). 


Approximately 80% of FYSO's work f5rce is engaged in CDA 
ent 


Ber avities. ASH sane Ont ator That FLOR ues 
expended in four Uniform Automatic Data Processing Systems 
(UADPS): ShesUn =o = eROP Mo ySten. fO>. Invenvomy Con =ol 


Memes  f0TCP), the Uaiform ADP System fer Stock Points 
(UADPS-SE), the Level II/III system, and the Disk Oriented 
Suppiy System (DOSS). heelmse Oo “heeuser sites -fo2 Sach 


System appears in Figure 3.1. 


Be. ORGANIZATIONAL STRUCTURE 


VCE howe do naceG soma Seuaccure? Of PTS). 
Gegeon: Pio aBea> pS -art sfuaee* ons such as 
; Pes ee ao Na 21 nee Navy budg- 
e 


t q a 
(ects On es odiec=2on SUDDSE., - S7e jc 
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control, s«andards development, dita base ad@enistration, 


training and ADP operations Supoort. These are the 
Comptroller Department (Cod2 91) and *he Management 
Department (Code 92). Ens SOfeWar]e guelity copemel branch 
Memos ene Planning divisian of Code 92. (Figure 3.3) The 


Ccenptroller Department also performs external missions 
peeruatng stock fund buigeting ani direc fleet support 
miiict LOns. 

The Operations Analysis Departnen= (Code 93) is «the 
Naval Supply Systems Command's (NAVSJP) eetrec2 oa) Agent or 
conducting analysis in logistics maiaagement. This d2part- 
ment is made up of operations research analysts and mathena- 
ticians who use various mathematical, Stasistical and 
Seconomic analysis technigues to study and improve ths 
procurement, financial and inventory management functions 
throughout the United States Na 
provided for all NAVSUP activitie 


Naval Operations and Chi2t of N 
Ss 


ve 
Vy. These services aré2 also 


P “he fleec, Chie sr On 
systems commands and varisua 


ce tHE CDA 


"A central design agency is d2fined as a singl2 organi- 
zation which designs, develops, implements and naintains 
automated data processing systems in support of multipls 
Operating sites." (Rer. 18] Piece) ne sO eCDAws DrOdUG= 2On 


Meweartiments (Code 94 «hrough 93) ar2 the lin? organizations 
d 


which are directly responsible for the development and main- 
*enance of standard ADP systems. The personnel in these 
departments are functional syst3ams designers, computer 
Sys -ems analysts, computer Specialises. and cOmputer 
programmers. Rhotr woe <, | devetooN=-n2 asd COCAMeS-az=len OF 


rh 


these programs, is *he major product of tne CDA. 
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Three basic principles necessitate the existence of «his 
mee OL production organization and directly impinge on its 
effectiveness. 

ge There must be @ potential jroup 
m@ntech perform a mission of functional sin 
with business volume of 2 magnitud2 suffi 
acguisition and operation of automat2d systems. 

2 Pies Sune=s2onal simiiarity s£ the individuel sites 


within the customer group nust be complete enough +o permi 


i 


pce gmece Of System Standardization by which the single 
product of the design agency can adequately support «he 
needs of mulitipls users, thus the cost of system develonoment 
and maintenance can be defrayed by the benefits obtained by 
tha many users. At the sane time, a marked degree of stan- 
dardization and improved management is obtained. 

oF The concentration of systen design and development 
Beem an a CDA affords s2portuniti2s for single operating 
sites tc obtain development of syst2ms that they could not 
afford to develop themselves. 

Bae wo geccives of a CDA is as £5llows: 

wo=7T> inttiet>s ADP develoonental action 


on rojects Phoshe have Uclaecztgone  “Sost 
benefit anaiysis and were determined = 


t 


i 


have a high ratio of benefit to cost. 
=—uor =MSures sort inued compat bibs-y of 
atic yS-eMs ws>h approve im ni litary stan- 
dardizat-on Be eet amd. =<2scing Suoply 
and financial management policy. 
- To optimize resvonsivensss «tO Logis=ic 
Memage:s 26) 2h= Elect and Snore es7an- 
lishments in the development and mainte- 
nance of assigned syst2ls. Ope = 7 un 
esponsiveness is tha timely production 
Ore accura<¢ Sepores odes ina) Ss scoc l= 
feces Gaqui ted =>. 2 Iprove, ans efs2c 27 s- 
ness of supoly, fa aanc iad and 
Maintenance funstions. ; 
= 0 emphasiz> user ee 8 Sele) tue Gis 
Savings ina Seyi L etepp ADGwsPaldwate, plan 
equipment 3: nd inventory iavestments in 
ae die yelopme nt and mnaintenance SE 
assigned systems. a 
MOM CnVOlVe USS, “Slices 2n 222 2den 722] 
Gee 20N Of IgG la <2 Olt) sOUCOz eur. =22s 
SSS shal Rneetine i 0 28 Bewlie sh onc and 
SEsnon- cs, pr.oricizatzon pi workicad, 
aml SUDDOE= df SYStSMS Prototyoling acd 
implementation. 





While 


See Sa tlLOn, 


Of subDpor 


serve che 


> ee, ade 
With design ae 
modularity gt al 
policy/procedur 
unique customer requitensats phage 
consideration of etliciemtly/tlexrp.l: 
trade-offs. 
=. 20 design and ea ADP mG 
which will be compatible wit 
oe ojected role of user sites duri 
uture years. 
poe lOgm PaEticupsate in the exchange 
information with other DID desiaqn age 
cles and to enhance systems effectiv 
ness and personnel proficizacyv. 


oe 
euro 


Sig 


Mes aisleuate 3 22) ,Project resdurce require- 
ments in the initial planning stage so 
ee oe ce cient lead tim2 iS ptovile 
fOr timely acquisition and develcpment. 
mao, paeDate CDA budgets Which Esflest 
sound and integcated production lans; 


to allocate resdsurces within the CDA in 
accordance witi reconcilsi budget/pro- 
die. 2on plans. 
ae ue, opt: mize <ODA Geo age ct aes Sad e- 
ture, staffing levels, and allocation 9f 
personnel ressurzes in oriar ‘to insure 
maximum DEGGUCENVLCY Of RYH PrioELty 
Pro do * pul 
© pursue personnel resruitment and 

raining prograas wien insure avail- 
ability Ce eadvanced Knowleij= andpskilis 
Peek St lss, data process: Ag ea Gait all 
management and pedated diss planes, | 
= enhance CDA productive capabili¢ 
phorugh | £n¢e use of special tools, 
ere Lu MNteontceave PLOGrammang, da 
base maha ement, DEceeotp: lense el 

ozther available ‘echnigqu:s. ve 
= To employ the most effestive training 
techniques available in ocder to imple- 
ment systems at new. us?r sites ind 
Bectall new asolicaticns 32 existing 
user, sites; +9 conduct 4 pdrogramn of 
£ield assistance which assir2S continued 
Sapparted 2 of user sites in operations 
y CDA systems. 
= io wo lize standard high-level 
d= A ocuinelais languages ¢t> the  naxinaun 
extent feasible and to use assembly 
imanguages only where technical sequirs= 
ments unequivocally dictats." (Ref. 19] 


~~ Cv 
jeryel 


mee oc fhe CONS ars involves ermpes cil il yern 


=Hey ate sepetated ines Logical fEunc 
me Becauce of chis separation, she CDdAs 
Same customers. FMS) as a CDa is divided 


Sel lLOwing a-eas: 


tJ 
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This department is responsible for the design, development, 
implementation and maintenance of environmenital systems 
software in Support of NAVSUP-soonsored ADP systems, 
meewmerng UADPS of stock points, UICP, the tridant pregras, 
the international logistics progran, and programs that are 
assigned. This department also performs these functions for 
the systems maintained by the other CDA Ys. 
Telecommunications networks sponsor2i by +th2 Naval Supply 


Systems Command are anothir ar2a in which cod2 94 is respon- 


sible for the environmental systems software. Pies depase— 
ment is made up of 109 somputer specialists and 27 peoples 
wh> handle all managerial and clsristal activities. Other 


major projects either designed or susvorted are: 

Meese Licss— stock point logistics inteqrated communica- 

tions environment 

Pepe = logistics data communications 

Be OLA = "On—iine autodin 

G. AUTODIN II - automatic digital network 
Stock Point Systems Desian and Procedures Departmern= (Cods 
gay, (F2qure 3.5) 
Gass depertmen:'s purpos> is +e dsvelop and naintain the 
automated systems for Navy stock oodint support including 
trident Logistic Data System (LDS), NALCOMIS, Automated 
eady Supply Stores System (ARSSS), Tape Oriented Supply 
Socom (TOSS), Disk Oriented Supply syseem (DOSS), SLeace Sac 
Be tT2 ct Sale, Legere Lie, Navy Automate P2ean Spo=24 200 


as 
Documentation System (NAVADS), Navy Automar ed Transper zation 


Data System (NATDS), Rea suCr at eonmOcesareons | Persona: 
Property Standard Syst3 (LIPS ik, Navy Tee sgeekoa 
Seocage-Toacking and Retrieval Systen (NISTARS), 


K 
Meters ar Monitoring and 2xvediting (RMNGE), cleseca Ls 
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FIGURE 3.4 Code 94 Organization 


SOURCE : 
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82 ANCH 
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SR ANCH 9424 
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MANAGERS 


ENVIRONMENT SYSTEMS 
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OPERATING SYSTEM 
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BRANCH 
9432 


AOP SYSTEMS 
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82 ANCH 


GENERAL ACCESS 
CONTROL DEVELOP MENT 
BRANCH 


9434 
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FIGURE 3.5 Code 95 Organization 
SOURCE: FMSO Organization Manual 
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Aeronautical Management Program (CLAMP), and Defense 
Warehousing and Shipment Process (DWASP). This department 
also assists customers with thea implementation of these ADP 
systems through development Of (Erasing dscunensas2on, 
initial training and installation assistance, monitoring o£ 


performance under operational conditions and follow-on field 


assistance. The department is involved with approximately 
eeNevy Stock points located around th world. 

Inventory Control Points Design and Procedures Department 
feoqge 896) (Figure 3.6) 


The purpose of this department is to develop and maintain 
the ICP*s UADPS design and work on refinements to these 
Peegiaam= tO Carry out NAVSTP and hardware SYSCIMS inventory 
eearro! functions. Theic principal customers are ‘the two 
Memor Navy ICPS: the Ships Parts Control Center (SPCC) and 
the Aviation Supply office (ASO). This department also 
develops and maintains d2tailed systams design for trident 
and ship-support functioas. Tos CCMpa2tsce (Of Aaperox:- 
mately 250 people and is 2 functionally orientési deparcument. 
eee nencial Systems Design and Provedures Departmens 
Pe) (Figure 3-7) 
This organization is responsible for systems design, devel- 
opnent implementation and naintenance services for headquar- 
ters, Naval material commani; eh22f Soe Naval Meterzal 
designated project managenent offices; and other parcici- 
pating headquarters commands and offices. It provides 
Bemevecs tO both of the najor customsxr groups; 2nVen~OLyY 
PieesoueoCines and stoOck points and othe> activities under 
the DICP and UADPS prograas in the 3 
meey cOn-EOL, stores accounting, disbdur 
Baycol!’ and ovérsonne!l accounting. 

The systems designed oy this organization support<s 9a 


MemetoeNeyy's financial jnventory report requizements, 75% 
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of the current Navy dollar resources under its ras 
Management system, ani 653% sf 189,000 civilian employee 
Salaries. 


Code 97 provides similar servi 


() 


>=s to the Wavy regional 
finance centers and evaluates the performance and develop 
such projects as the Intagrated Disbursing and Accounting 
(TD A) system, Standard <Accountinzy and Reporting System 
(shee) and the Automat2i Procurement and Accounting Data 
Entry (APADE) Systen. 

awe department consists of ‘three nilitary officers 
pee vatean complement of 244, Sov2f2ne@ the fulleeira-gqe of 
financial systems and data processing expertise. 
international Logistics Support Department (Cod= 98) (Figures 
bo) 
This d¢partment is responsible for the maintenance and 
continual enhancement of the Managenent Infornation System 
Peameeteenna-lOnal Logistics (MISIL). Its principal customer 
is the Naval International Control Jfficer (NAVILC3) which 


utilizes its systems tc provide services to numerous allied 


navies and governments. The departnent handles complets 
automation for the Saudi Arabiar's Navy supply system and 
micetmars=on of support Systems (supply, SHRVL=Eonnertay 
personnel and financial) for Kuwait's Navy. It establishes 


femetenag programs for United States Navy Supply Cozps 
personnel going +o Military Assistance Advisory Sroup (MAAG) 
miemeand aevelops an advante base sugply System £95 Overseas 


Supoly depos. 


Bemeeotores DESIGN AND QUALITY ASSURANCE PROCESS 


item cop down design metnod 1s used as =he s 18 
Seoneadeh £O> new System/program d:velopmen= in the FMSO 
environment. This appcoach is alse known aS stepwiss 

n 


Peereremen=, hierarchial design, levels of abstrac*ion 


ub 
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design by explosion. The nethod uses a breakdown technique, 
@ividing the main function into smaller subfunctions. The 
Pemmaayesunction, thought 9f as the santral or driving func- 
fuom, =-.5 designed first; then steowise, this process is 
Continued until the smalles* functional unit of the systen 
is specified. 

Because of this breakiown, +th2 system can be viewed as 
modules. Every stage of the syst2m and program yields 3 
ese bl= ou~put. Each subsequent subfunction which is 
defined becomes a modul2 of code whith, when tested, serves 
to retes* and more thoroughly test all higher lavel modules. 

The use of hierarchical charts forces the design cf new 
System/programs in the top down method. Phesmise Of8Visual 

theicz 
n 


ani not their 


diagrams shows the major functions and subfunctions 
with the emphasis on their suborijiinatio 
Hogacal flow. 

FMSO personnel state that the system designers focus on 
what is required and the systems analysis workers focus on 
fou =O achieve it. The system designer, workin yer 
closely with the system aser, defires what information is 
meainecd, now it is tegquirsd, when it is required, and for 
whom it is required. This helps tremendously in keeping 
this precess of development at minimum cost. 

The systen developmeit process is deliniated in FMSO's 
CDA Manégement Handbook. Appendix A, taken from the hand- 
book, shows che process. 

0) h eeiaepa. *he development process a quality 2ssuran 
Baocktsst is required. P O 
checklist. 
al 


On 21 January 1982, 2 qual: 
was published. in hes  Sepor. 


ewe ogGham =aSk G2oun Tepos= 
S 
meen the f€ollowing: an internal 


S 
an ¢xXamination of the ADP dGavelopnent nodel and *he CDA 





QUALITY ASSURANCE CHECKLIST 





Program/version Date 
ELEMENT SIGNATURE DATE 
1. Scope of Release: 


a 


ae 


4. 


a. New program/complete 
rewrite 

b. Major modification 

c. Moderate revision 

d. Minor adjustment 


Criticality of Release: 

a. Mandatory (HQ. directed) 

b. PTR response 

c. Solves serious program 
deficiencies 

d. Highly desirable 
enhancement 

e. Routine release 


Urgency of Implementation: 


a. Immediate 
b. No later than 
c. Optional 


Level of Testing: 
a. Local FMSO testing with 


simulated test data 
5. Service tested at 


¢. Prototyped/Op Reviewed 
at 

d. Tested by FMSO with live 
data files/transactions 
from 





FIGURE 3.9 Quality Assurance Checklist 
SOURCE: FMSO Quality Assurance Program 
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ELEMENT 


Meets standards of hardware 


utilization 


Availability of proper hard- 
ware verified at user sites 


Impact on hardware capacity 
assessed and verified as 
available at user sites 


Release will lengthen real 
time responses by 


Documentation meets standards 
of NAVSUP Pub. 506 


(Rev. April 1976) 


a. Functional Description 


b. System/Subsystem Specification 


c. Program Specification 


d. Computer Operation Manual 


e. Program Maintenance Manual 


f. Test and Implementation Plan 


g. User's Manual 


h. Data Requirements Document 


1. Data Base Specification 


j. Change Transmittal Notice 


k. Test Analysis Report 


1. Project Manual 


m. Technical Report 


n. Technical Note 


System/Subsystem Specification 
was approved by NAVSUP 


SIGNATURE 





(0 I 
o 
ar te > 


DATE 


FIGURE 3.9 Quality Assurance Checklist (Contd. ) 
SOURCE: FMSO Quality Assurance Program 
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ll. 


has 


13. 


14. 


ioe 


16. 


17. 


18. 


20. 


2 he 


ELEMENT 


Satisfies System/Subsystem Speci- 
fication as Approved 


Satisfies Program Specification 


File Integration/Integrity 
Verified 


System Intcegration/Integrity 
Verified 


Tested in (Simulaced/ 
Production) Environment 


Test Data Base Updated To 
Ensure Adequate "Real World" 


Cases 


Program Restart Capability 
Verified 


Program Interfaces with Software: 
a. Currently Implemented 
b. New Sortware Package 


Required 
c. Scheduled for Release 





(Software) Release is Upward 
Compatible with Prior Releases 


Programs have been developed, 
anaiyzed, coded and reviewed at 
critical steps utilizing the 

FMSO standard Improved Pro- 
gramming Techniques, as described 
in FMSOINTINST . 


User Training Has Been Provided/ 
Is Not Required 


a. Type Training Provided 


Db. Date Training Completed 





SIGNATURE 





EEE 


DATE 


a 


FIGURE 3.9 Quality Assurance Checklist (Contd. ) 
SOURCE: FMSO Quality Assurance Program 
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22. Standard data element names have 
been used throughout the program 
coding. 


23. Remarks/qualification/explanation: 


24. Element certification responsibilities: see item 24, enclosure (4) 


for individual element certification responsibility. 


25. QUALITY ASSURANCE CHECKLIST CERTIFICATION. 


Each of the above quality assurance checkpoints has been verified/ 


validated by myself or by persons under my supervision. 


The responses 


given are true and correct to the best of my professional knowledge. I 
understand that individual quality assurance level is a significant 
factor in each annual performance rating. I certify that this program 
release has met all FMSO quality assurance tests and standards and is 


ready for release to customer activities. 


Branch Head 


Division Head 


Department Head 





Date 





Date 





Date 


FIGURE 3.9 Quality Assurance Checklist (Contd.) 


SOURCE: FMSO Quality Assurance Program 
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handbooks; a review of the functional operations of the 
@@aitety control organization, research and csview of th 
technical libraries and oublications dealing with software 
quality assurance programs, and an external survey question- 
naire directed to the FMSI-systems cist 

W 


omer community. 
ieee por. Stated that the following factors in the FMSO 
envirenment prejudice quality in varying degrees: 
iio etiandatsd, HUle pe and dissimilar hardware 
Seviigurations. 
2. Unrealistic/inflexibl e/mandate De Onjiace completion 
dates. 


3. Ill defined or undocumented rezuirements. 
4. Inadequate test facilities. 
Seeemunaeng (budget /travel) con 
Se SOgeGeePELOritization pro 
7. Diversity of customer astivity in systens/processing 
requirments. 
8. System changes/controls edicted from iAgency/systen 
command echelons. 
were deral DrocUursmen: policies and regulations. 
The *ask group's work 2xverience, 2 review of industri 
J2tereture, and «the internal survey revealed thea 
molrowing specific conditions 2xist: 
1. Poorly Defined Requicements/Sp2acificatzions 
a) FMSO design procedures/practices tend to be apnoli- 
Cetaon-oriented and at ths disezretion of the 
developer. 
b) System design and analysis knowledge is no* being 
shared between or within ths CDAs. 
eumcoemal cevidw sand walktheoudqns same ee. 
carried out propecly during system development. 


Gyeernecre iS no @aisible interaction weeh customers. 
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e) 


f) 


g) 


h) 


System analysts are not always required during unit 


sesting. 

Pee cian e Keeps 5n Of the orogzam trouble =epert, 
there is no provision for Pec or consoli 
dating customer feedoack information on 2 recur- 


cing basis. 

ADP system developmental information ard e¢xperi- 
ence is not formally or consistently shared among 
developmental organiza2 tions. 

A more business-like, couprehensive policy and 
procedures document is necessary for FMSO/customer 


relationships. 


Unrealistic Schedulss/Estimatei Completion Dates 


a) 


D) 


Cc) 


qd) 


Insufficient Testing Time/Test Facil 


a) 


) 


Mandatory due dates caus? abbreviation of gualiity 
events. 

Completion dat2 as set by th2 POASM is usually "set 
=n stone." 

Peoject trackinj/status reporting and xresourc: 
Seceuntang arte not currently provided on an inte- 
grated basis for project maiagement. 
feveseC ise lingets=d automated scavdabilicy in the are 
Semgocumentation oO 5o0aration, S-oRaG Ss 


packaging and distribution. 


log 

ieee ltap diy of hardware (F4S2), basically the test 
beds, orecludes estimatinjy realistic time frames 
and comoletion dates. 
Wiese s J2CK Or UnZiEOrmicy, in che assigqnn 
= (oeof aac. GES OS VS’ Sivas s ate DDIGraM/SYStTSN 
osc Lh Gi. 
Nomunmirorn meechosds Or peosceduce exist fcr estad- 

S 


ism 1g and Maintaining F45)* 
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d) An undisciplined approach t> program testing among 
CDAs is used. 
SmpoO@seWeoo Sngine=cingetc n3t a distinct function. 
4. Lack of "State-of-The-Art" Davelopmental Tools ani 
Aids 


5. Urnecessary Paperwork and Procs2sses 


Bee «CONCLUSION 


As shown in the system development process, Appendix A, 
the quality assurance branch interfaces with deéevelcpment 
Bemeormes in tracking of the functional description andi 
system specifications ani in checking the product before 
release for compliance with standaris and quality assuranc2 
procedures (check list). All tests and project reviews are 
carried out by the developnent persoinel with the use cf tha 
quality assurance check list. The actual duties of <*h3 
quality assurance branch aay be viawed as only administra- 
save in nature. The next chapter shows how other quality 


assurance groups function in their srganizations. 





IV. INTERVIEWS 


This chapter presents the authosr-conducted interviews 
Mescieepersommel of <he quality assurance groups in «he 
computer organizations addressei in Chapter ee The 


following questicns wer2 2sked duriny the interview: 


1. 


2 


Bie 


4. 


a 
« 


ticmesaages 2S en join3i 


Where does the quality assuraace group fit into thea 
organization? 

What type of authority/powéerc does the Calaiaas ¥ 
assurance group have over the software product? 

What qualifications do the people in the quality 
assurance group hav=? 

How does the guality assurance group interface with 
the design/development group? 

Wat tools, methodslogies, Ir techniques does «he 
quality assurance group use to do their job? 

Are historical records of orosblems with softwars 
products kept after the products! release, and who in 
she company's organization ketos them? 

Who handles problems with software after czlease, and 
how are such problens handled? 

iieedussbramd Mew product is designed, who in <he 
company's organization trains the customer on this 
Be cdaucs 7 


compare «he interviews with 


326, 
Browa2scuss20ns in Chapters II and III. 


Re 6 ©HEWEBTT PACKARD 


The Hewlet*+ Packard Company is a major desigrer= and 
fe) 


Manufacturer of precision 


do 
ment, 


(D 
— 
iD 
‘@) 
ct 
if 


tpmen* for measure- 


n u 
analysis and computation. Th2 company makes nore «han 
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PPocoe Pesauet= wWhicn ar2 sold worldwide and have broad 
application in the fielis of science, engineering, business, 
mpauSstry, medicine and siucation. Pheir 2£0UF Main produc 
segments ars: 

ee Peece=enue Mata PLOduc +s -= somputational products 

luding personal computing devices, desk too computers for 
pugere=r2ng and scientific applications, Small business 
computers, and lérger computer systans for both business and 
technical needs. They also offer a large selection of 
application software and have developed a wida selection of 
peripheral equipment for use with theic computers, including 
computer terminals, disc u3mories, printers and plotters. 

Ae Electronic Test and Measurc2neant Products -- rang? 
from general purpose instruments ani systems for electronic 

[and Meastbement tO specializ2i instrumentation for 
conputed measurements to components and accessories such as 
Microwave semiconductors, optoelectric displays, bar code 
readers, and fiber optic systems. 

She Medtcsemey CCebOnlLsc EQUipment -=- gamily sf nore than 
SvugemMedi cal PEOGUCtTS Which ara used For diagnosing, sacni- 
Beene nd ee =Sede=' ng Paci ants, ana for medical information 
management. This equipment ranges from por«abl2= electzrocar- 
diographs to powerful computer-a1id2i patient monitoring and 
patient data management systems. 


4. Analytical Instrumantatton -<-- Product family 


3 
includes gas and liquid chromatographs, mass spectrometers, 
peeoudenecer lid samplers, analytical laboratory data acaui- 
SPeceomwrcyscess, and sp2ctroohstometers. ris 


Seameacs Seseancn, » SPrOdUcti on, end eavcsconmen al 
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Figures 4.1 thru 4.3 


ture of the Hewlett Packard Company. 


“here is the technical computers 


Systems Division is a part. The 


ance organizaticn comes from this ii 


Peomets Pot Only responsible for 
but also for hardwaré 
meoagmes reliability, 

Tegulation 


Beoduct ion and safety, 


assurance engineering group 


the education and experience to 
programmers themselves, 
Their 


Peoduct GesSigners 


assurance. main purpose 


from the 
assi 
Face between designers 


Bie mor ell areas of Yewlett 


Gammany 2S TOVing in that diresti 

The quality 
@@ienOonrity over the product. 
if they thought the product was 
att 


their abriity 


released. They state 


reputation and 
fewer e and i occurs, the 
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2. Quality Assurance and Design2r Interface 


Figure 4.4 shows the development cycle as it is 


perceived by Hewlet*+ Packard personn:l. When the designers 
from research and development hav2 an idea for anew 


product, a proposal is sent *o management. If permission is 
given, a product design group is forned consisting of people 
from marketing, research and davelopnén*, manufacturing, and 
quality assurance. When the c@sign its laid out, the quality 
assurance people ask "What if" gqisstions *9 ensure all 
aspects are considered. The company sets nd particular 
Specifications to which the designers must adhere, so they 
have the freedom to be craative. fh main languages used by 
the designers are assenbl2r, Pascal, and FORTRAN because 


their vroducts tend to be more techiical than commercial in 


Hat ure. They also produce environnental and applications 
SOE-wWare. One person from quality assuranc2 is assigned +o 


each project. 

During the requirement phase of the dav 
process, an investigation has to be completed i 
produce a detailed specification plan and a us#r interface 
Spsciticas:on plan. In “*he external design seam 
quality assurance people must produs2 a quality o1 
néating the quality goals 9r objects of the project a 
they are to be measurei. Titcmess app eOnlem aeea = 5 
quality assurance people because 1f the product is generat 
az a customer's request, the request is usually not specific 
Or incomplete. Pome NeCs eae tae LOLMaliz=d Conmunica- 
tions be escablished to aliminate this problen. 


h 
In the internal design ohase of the development 
= 


Greme, oe -ne internal specifications, POpedowrecec2gi7- and 
Submodule design take olace. The quality assurance 
: : iaws 


Bepoeave set UP, NMoOnitor, and partiscionpate in design <r: 
and codé revi 


= 
Moms teoce Oral cest plan. 





| 
| 
| 
| 
| 
| 
| 
| 
| 
| 
| 
| 
| 
| 
| 


| 


PROPOSAL OR CONCEPT 


gE ED ee ee 


REQULREMENTS 
l. Investigation 


2. External Desiyzn 


DEVELOPMENT 

Ll. Internal Design 
-. %Implementation 
BDeweincerracien & Test 


A. Functional 


eee ieee 8 ae EE ee ED ep EE EES SSE ome ee on ep SP ae, ee ee ee eee SP La Oe awe ere OS eee SE eee ee oe core ee ee ce ces gg OE Lee ote Oe Lee ae eee OE ae Te arate SE enter 


7 
PO ee > Si ht «E> as! > = GD qa a OD > i> ote CSG ase oO o ae OR ae) cpa fee 42h ae eae sD ote ee ae Gee ee eo sie 


Systems 
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Oe rinieactur ing 
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Pigyure 4. 4 Hewlett Packard Software Developnent Cycle 


Source: Interview of Hewlett Packard Sufttware QualitY Assurance 
Personnel 
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During the implementation statement, the quality 


fu 
i) 
0) 
= 


rance peopl? set ap the systens test plan. Actual 
testing is not accomplish2d until the integration and bast 
segments, and it is done on the function and systems levels. 
Although the functional *+sst plan is produced by the qualit 
assurance peopla, the actual tssting is done by ths 
designers themselves. This level of testing is viewed 
mainly as a debugging 2xarcise and would be a waste of ths 
Guality assurance organization's tim= and resources if dons 
by them. Act the systems level of testing, the plan and 
tests are done by the quality 2ssurance people. These tests 
ace viewed as a third party auditing inspection of <«hs 
peoduct. This «third party testing is done because Hewlett 
Packard does not believe that the program d2signfrs an 
analysts can be completely objective about their product. 
The quali*y assurance groupis alss responsible for «hs 
Dasimagerg Of all test plans for reusability. There are no 
percentages of correctn2ss sought during these t+esting 
levels. When this segment is complete, Phas ocroduces is 
eonsidered 100% correct.. 

According to th? quality assurance people, ancther 
problem area is the schedule planning. The designers do not 
*hink that problems will occur during this testing phase, so 


they have to be carsful 9 plan for axtra time if problems 


olele se 
Meeeceeene MGhaliay COrpeification segment, which is 
Paceiedtiyea cUstomMm=> accsoterce inspection, and «he produc- 
Po o2icesen fica. 7On Segment, comes =h= manufacturing segment 
n 


n Seistde on =e produce £5 
Meee et a CUStOM=2=c° requested the preduct, all thea 
e a 


Sc manuals and 2ny other 


SI 





3. 


bel: 


tnvolvement with its software 


Hewlett Packard 
not abandon their customers 


software 1s copyrighted so if 


after sales. 


eves in Meradlenm co Grave" 


products, which neans they do 
All Hewlett 


any problems after 


Dackari 


there are 


e 
S| a 
_ ww 


tha 
unless the 


is in operation, sost 9 the customer is $100 per 


Meu= for repairs Custorer has a subscription 


Service. Subscription service entitles the customer to havea 


ae DOWiere mao. 
abe 
updated version is 


hs 


the customer's system 


sofware repaired, updated, or replaced 


by 
for any customer, 


= 


5 We Cl is 
the 


have 


This service includes plan a progran 


Weeaqeed Or fi xed 


sent out to all other customers whe same progran. 


is left 


hd , 
7 - 
<- 


The décision ¢o use Cueh en 


momene CUStTOMEr. 

ieee re iS a proplen, notifies 
the field alls 
around to keep *he customer! 
neom the the 


manufacturer, and eventually 


aAGCzIveee ya which, neces NWOrK 


a 


= 


Sad 


program 


fecord actayaty, problem is 


Via SUPPOEt, 


research and development who design 


ie 


Benors<~1ze ~he pzoblem ani place 


2+ is eventually fixed. NiOomen. Som! 


; 
ec 


changes *o programs ar2 maintainei. 


The quality 2ssurance organiza*ion 
eld a 


£e 
= 


ideas and chanjy2s in this 


HOw ripssVe =tS DOLD Gren. 


“hs 


Fowlotr 





Be. TRW 


TRW is a diversified nuitinational company which manu- 
factures a wide range of products from compon2nts for cars 
anid trucks +9 defense 2zlectronics and space systems. TRW 
produces transistors, resistors, capacitors, disdes, poten- 
tiometers, *rimmers, tuning devices, TV convergence yckes, 
connectors, transformers, paTeodacwseli= “DCabds, electric 
motors, electric data processing terminals, and jet engines 
parts. Other products include pumps, fluid handling equip- 
nent, nuclear reactor cd) mponents, EeStiexs, bearings, 
mieerng <lOlsS, and hand ts>I1s. 

This company handles defeansé systems contracts which 
include the development of software and *the construction of 


the entire system. 
1. OFga 


TRW is divided into many groups because of its 
diversification. One of thes® groups, the défense systems 
Seop, Con<2ins the enginzsering divisicn o£ which the prod- 
ucts assurance organization is a part. (Figure 4.5) This 
level is made up of managers whd are assigned to ‘he 
different projects in assistant project managér capacity. 
This department is not just concerned with softwares product 
assurance, but also with hardware ani system engineering and 
design (SEAD) product assurance. (figure 4.6) 

Figure 4.7 shows the standacd work breakdown struc- 
ture for any oproduct in the dafenss systems group as it is 
concerned with product 2ssurance. The assistant projecc 
Manager heads up a staff xf personnel who work in the areas 
Semoroieety ascurance, Conti guration nanegement, relianbility, 


and safety. 


Pecieemte CShOWs the Standard work breakdown struc- 
Sirsa OmeNci= GUali=7y eassurance area of the project which is 
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further subdivided into management, software, hardware and 
system. 

When working on military contracts, the company must 
BOomlOow Specifica-10ns required for csontract award. One of 
these specifications is “MIL-S-52779A "Software Quality 
Assurance Program Requirements" of 1 August 1979. This 
document states the requirement for the establishment and 
implementation of a software quality assurance program. pes 
2s hoped that this program could b2 tailored, #zconomically 
manned and developed in conjunctidn with the contractors 
peograms of this type. Pie COMtractor 4S required to docu- 
ment this program in the form of a software quality assur- 
ance plan which meets its specifications. This plan has to 
misweley O©dan=Zational respoensmbility and authority for its 
execution and make timely provisions for special needs 
femterols, {OOls, fecilicies, skills, stc.). Because this 
Meamercdae Or =~he COMitrace, §%t 2S considered to give the prod- 


Weesvassurance Otgani zation 22s authostity over the project. 


ndard dutias expected ¢5 be performed by the 
personnel in the managemant area sf the project are as 
iG 


follows: (Figurs 4.9) 


ae elon ning ands contro. 


Sie GO Oroy  emdeseeceien and participate in +t 
fetieamecscn Of qualit 235su n 


BeonCom=a dD. | =heO +] 


@ 
'O 
4 


Ss O 
miolemen=a= "On DPlamypeprogs=ct schedules, documentation ola 
ani other sinilar documents. 

(2) TO defin= Sn= cualzty assurancce ta 
Loci] —2pDEODELace Derscnnel. T> monitor their perftora- 


ne 
ance and prepare status reports. 


Or 
Or 
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PLANNING 
AND CONTROL 


@ PROJECT PLANS 
@ QA TASKS 
e CONTRACT CHANGES 


QA PLANS AND 
PROCEDURES 


e@ QA PLAN 
@ QA PROCEDURES 


PROJECT 
INTERFACES 


PROJECT MANAGEMENT 
PROJECT PA MANAGEMENT 
FUNCTIONAL PA MANACEMENT 
TRW FUNCTIONAL MANAGEMENT 
PROJECT BOARDS 

QA OPERATIONS 
CONFIGURATION MANAGEMENT 


CUSTOMER 
INTERFACES 


AFPRO 

FORMAL REVIEWS & AUDITS 
ASSOCIATE CONTRACTOR 
DOCUMENTATION REVIEW 


SUBCONTRACTOR / 
SUPPLIER MCMT. 


——_ 


e@ SELECTION 
@ REQUIREMENTS DEFINITION 
e MONITORING AND CONTROL 
FIGURE 4.9 TRW Quality Assurance Project WBS - Management Detail 


SOURCE: TRW Status Report on Standardization of Quality Assurance 
Functions Task, 20 April 1982 
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(3) Momnenitotr all actions in conjunction with 


contract and engineering changes. 
be. Quality Assurance Plans and Precedures 


They are required to direct the generation of 
O 


c 
the quality assurance plan which follows the controlling 
government specification MIL-S-527794,to review, maintain, 
Pissed -2 12 throughout she project's life. This plan is 
required to address: 

(1) Tools, Lechnagque Ss; mexhodologies and 
records to be employed in *ha performance of the work <to 
support the quality assurance objectives. 

(2) Procedur2s by which design documentation is 


reviewed to evaluate design logic, fulfillment of requite- 


ments, complsteness, and compliance wit specified 
standards. 
(3) Contractor's procedures f9r formally 


moma yes oNO> COrtizyirg =_h2 descriotion, authorization and 
eempletion of work performed under contract. 

(4) Documentation of standards, programming 
convenzticns and practices to be used for all software. 

(5) Documentation of tha ChebasGcon"s proce- 
@iaes ana Controls for handling of sdurce code and object 


@ee—c eng ctelated data in their various forms and versiors. 


(6) Documentation of conz=ractor's procedures 
for preparation and execution of reviews and audits neces- 
Sary in establishing traceability Seen eal CClezac. 
requirements. 


Coe =Oojece LNntssfaces 


The management detail aldresses the interfaces 


between voroject manager, assistant project manager, sud 
Peomeemmertagers and Others in Conjuiction with the voroject. 
Thay at<~erd the staff meetings and czspond *o action items. 
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d. Customer Interfaces 


The management detail works with the customer 
representative offices, hosts thei visits ani formal 


reviews and take care of documentation to and from «the 


customer. 
e. Subcontractor and Suppliz2r Management 


Figure 4.10 delinates the duties of the 
personnel in the softwara area of thea preject. The «three 
groupings are: 

(1) Management Support -- carries sut duties 

in support of the management section 

EieuspEo Toc -. 

(2) Sr@inessiig 

(a) Edemeteyoand d=fane =he quality 
standards and procedures that will 
be Followed ducing the design, 
development, programming, testing 
and documentation stages. 

(b) Identify software tools and special 
methodologies that would be used in 
performance of quality assurance 
task. Establish vrocedures for their 
use and ensure their use jiuring the 
DEOns ce. 


(Glee Co eerclpa ce Lorde fin: te 
Dah oer als> a D 


boards ena cu oards. 
(¢) Maintain recoris and files of documen- 
Saeco ew rot adherence =o 


* 


Standards. 


(3) Diez ae § Of Ss 
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PROJECT INTERFACES 


CUSTOMER INTERFACES 
SUBCONTRACTOR /SUPPLIER INTERFACES 


ENGINEERING 


S/W STANDARDS AND PROCEDURES 
S/W TOOLS AND METHODOLOGIES 
S/W PROBLEM CONTROL 

FORMAL REVIEWS 

PROJECT BOARDS 

RECORDS MAINTENANCE 
DOCUMENTATION REVIEW 


OPERATIONS 


AUDITS 
TESTS 
INSPECTIONS 
SITE SUPPORT 

















FIGURE 4.10 TRW Quality Assurance Project WBS - Software Detai!] 


SOURCE: TRW Status Report on Standardization of Quality Assurance 
Functions Task, 20 April 1982 
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(a) Perform audits on project activities. 

(b) Participate in seach level of software 
Pestrageas designed by the quality 
assurance plan and perform surve2illancs 
aC tiv it 2's. 

(ey Pertsrm visual inspections of all 
software products pvurchased with hard- 
wars from supplier. 

(ese Stomnmauallty 2assunamce function at 
each site and remote site for testing. 


PaecUuiemqd eny documentation audit, a discrepancy is found, 


the discrepancy is documented and is taken first to «the 
responsible designer. Pees a Ces2ain amount of time, the 
error is not corrected, the problen is taken to «he next 
level in the project organization. The problem will travel 
up the crganization until the discrepancy is corrected even 
if it méans going outside the preject's environnent. 


Approximately 2 to 5.5% of the entire project's 
mind=s2s Chat gedeso GUality product assurance, but it is the 
Opinion of the managers of quality assurance in the TRW 


company thaz the cost of quality assurance is Z2ro. 


Once a projuct has been accepted by the 
customer, with the signing of def2ise form DD250 Material 
Inspection and Receiving Report, the legal obligation of TRW 
is ended. If any problems arise aftsr release, the customer 


pays to have more work dons. 

RETSSERCS 

bes SOreaiu LN=E5rvView with ar. (Veeeieam: VV. Buck, Dre oduc= 
Assuzance Manager; Mr. Samuel &. Benesch, Department Manager 
PeoGueo ASSUramce: and Mr. Marten r. Kenehan, Senior Staff 
Engineer of the Defense and Spacs Sroup of IRW, Redondo 


Begen california Om 7 May 1982. 
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Figure 4.11 shows the structure of the [BM organiza- 
mot as Of March 1°82. iemsnewmse chat, Undsr  &he staff 
level, ehewmeempany iS divided tnto two najor areas, 
marketing and service and manufacturing and development 
Unier these areas, the grouping of Jivisions start in which, 
under the information systems and technology group, she 
G@evera. products division 2xists. 

The general products divisioai, with its headgquar«ers 
Meecated 2n San Jose, California, is responsible for the 
cdevelopmen* of all hardware and software products at IBM. 
It has two development laboratories, one located in Santa 
Teresa, California and the other in Tucson, Arizona. (Figures 
et 2) 


The general products divisisoi is headed by a presi- 
e)os 


den+ with a vice-president in charjy2 of each operational 
department including: hardware, sdftware, manufacturing, 
financing, support and products assurance. Heading ¢ach 


development laboratory is a center nanager with functional 
managers in charge of 2ach d2partmint below hin. Wee 
each of the development can*ers, 2 functional manager in 
G@ieaeg=) Of products Or quality assucaics. 

The quality assurance departnent within this organi- 
zation is completely indenoendent of sthe D ents. The 
software products developsi in these labo 
Shemeuy=aeMent Or @peead=ional tool area 
they are produced in all o£ the major pro 

lity assurance group does have au 
ucSS that are new and azz about «5 be announ 
dewaseeDe mg shit opel to customers. If this group 


h 
fee eeroe ogres thacmemepeoiuct is realy, it is cot released. 
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FIGURE 4.11 IBM Organization 
SOURCE: Interview with IBM Products Assurance Personne! 


19 





| 


GENERAL PRODUCTS DIVISION 





HARDWARE SOF TWARE MANUFACTURING SUPPORT FINANCE | 


SANTA | TUCSON 
TERESA | LAB 
LAB 


CENTER MANAGER 


MANAGER FOR DEVELOPMENT 


MANAGER FOR QUALITY ASSURANCE 
PRODUCTS VERIFICATION & 





ASS URANCE TEST 
DEPTS DEPT 


Gaeises Gea ® SE GS ee ee a (s_ ee aa Se a es See eae eS SO 


-—— 


a ee nt ee i SE ce A a ee ee eee ee 





| 
| 
| 
L 
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Source: Interview with IBM Products Assurance Personnel 
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Source: Interview with IBM Products Assurance Personnel 


74 





The decision for product cslease is no«+ driven by any other 
Bacstors. 

The quality assurance department is divide at 
ite cemde Visions, twWO Of which are orsducts assurance, ani 
the other is verification and testing. Every softwar: 
product developed is divided between the two product assur- 
ance divisions. The number of people assigned is a function 
of the project's size and their schedule depends or zhat of 
+he developers. At the end of the development cycle, ail 


magitees SO =hrongqh the ver ttication and testing division. 
of 


2. Quality ASSuransce and Design Interface 


en @np eae 2a a= =_=P a 2 os ep eee a = aaa ae @= as a= an @n @e 2p om 2a 2a 2a oe 


The quality assurance group interfaces with <+he 
program developers throughout the entire development cycle. 
(Figure 4.14) The peopl2= within tais group have no prere- 
quisite skill requirement and most aave varied backgrounds 
ranging from programming 2x pertise to marketing skills. Ie: 
do their job, they depend nainly on their experience and gut 
feslings. It is not considered necessary for them to have 3 
programming or computer eaginearing background because? i+ is 
very rare that they have t9 inspect the actual cod= itself. 

3 


€ 
Within each development d2partment are performance groups 
Pp 


(D 
rH 
i= 
UO 
Qu 


who ¢xamine the code and test it ically throughout the 
development cycle. 

The managers of the development groups depend on the 
people from products assurance for ‘their objectiviczy and do 
not view them aS a resource zcol. Th2s® products assutance 


Pee no mCoOLe- bits £0 the ocoduc* in Eh] following ways: 
asa Planning 


Before any work can be started, a projec* plan 
has =O be su= “«ogether in which the oregramners have <9 


Glaam which development style out 39 a vossible three will 
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be used for this project. [This plan is named the 
Comprehensive Evaluation Plan (CEP) which also takes into 
account the quality assurance procedares, use of resources, 
and the project's schedule. It is considered the main plan- 
hing document and has to b@? approv2i by the products 2ssur- 


ance division before the project is started. 
be. Early Warnings 


Tf at any tim? during the development cycle, «he 
quality assurance inspector sees anything which might kee 
the program development group from k2eping schedul¢, they 


Metery =he project flanager. 
ce. Value Added 


Dipeedueargeens pEOCeSS, the quality assurance 
people feel that something could be added to the software to 


enhance or improve it, they inform +h development group. 
des EGUGaAton 


The education of the orogrammers on possible 
development tools, whether developadi in house or externally, 
Mmemcasssec CUT bY this DJEtJanization. 

IBM sets standards requice: 


m 
PUwireeticouwche produc =s, bu= <chere is flexibility in their 


Or 


Weemoccatse tt) is let =o the discretion of the programmer. 
Diewvesaracaeson “and t=se="q people carry out 
MieesemNanG=2ONal testing ac the ead of the development 
beeGecs, PCr cOrmIng DaSitally user oftented t2sts. Their 
main objective is +o debug these products 9 any user 
Oriented vroblems. 

oroduct assurance, performances 

S 


S 9 
Srowepwesc Verification and tes: groups interface 
St % Vv 


b= 
{* 
E06) 


Seg Camron ifsiaing Quality produc <s. 


is AL 
=e ae Oe her b 


Ts 





A Review and Inspection process (R5I) 25 GCauricsa Cut by the 
programmers themselves throughout the development cycle. It 
is carried out either in 2 formal manner in which a meeting 
is held with the programmers and 2 moderator and they 
discuss the program and its progress in depth, or it can b3 
held on an informal basis with only the programmers! imnne- 
diate peers present. A representative of the product assur- 


ance division is required to attend these meetings. 
S- Operations 


Cnce a product has been released, the field engi- 
neeting division is responsible for rémedying any problems 
experienced by the customers in use of the product. ay) suits 
@2yision is also responsible for maintaining a2 historical 
racking record on probléens with the software produc*s once 
Beeche field. Tf a projuct is to be renewed or enhanced, 


the products assurance p2aple can ceques+ this historical 


information, but they are not required to keep track of it. 
If acompletely new oroduct is released by thse 
@enpany for Which the users would require «training, the 
responsibility for this training is 25sumed by the marketing 
division. Requests SOG. NEW DE.duc= are not received 
directly by the development laboratories, but through the 


women LBM user groups, SHARE and SUIDE, which mec+ twice 
yearly to discuss problems and possible ideas for new prod- 
Wer S. The marketing division is also constaatly carrying 
Suemetey-vs 2: Customers fOr new product ideas. 

The people of the quality assurance department 
mrouGnowena. *heiz> main objective was tc maintain a wide 
renqe perspective of ‘*h2 vroduct isvelopment process and 


never to become overly involved with datails. 


Z8 





Reference 


Personal interview with Mr. Barron oa MepOnald and Mr. 
Norman Towns of «the products assurance GaCcup, I BM 
Development Center, Sateaehecssa, california on 21 April 
1982. 

De AMDAHL 


Amdéhl is a high terhnology company engaged in tha 
state-of-the-art design, developnent, WEL, Obie (ere hm se alra 
Macketing and maintenanc2 of large mainframe computers, 
software and communication systems. These preducts are used 
by large computer users ia tha full spectrum 9f commercial 
and scientific data processing ¢environnents. 

The company's central processing unit's design strategy 
@s)<O Locus on che develoonent of sfficienz= design architec- 
[aee tor high pezformance, dapemdapie eve “anderisexi bility 
Bebeazdture enhancement of the product. 

The company's communication syst2ms division designs and 
Menufactures digital conmunication networks whic allow 


users *o interface with multiple g2ographically dispersed 


systems. 
A@denl also offers a number of services ZOe ees 
customers. nie =Car diana Oc Stoss — rainaing support 


with specialists in both hardware ani softw 


QO 
" 
2 
+4 
cr 
ra By 

+ 
ray) 

4 
}— 
O 
t4 
(D 
er 


There are also expanded 2ducational offerin 
Sea ainge =o enhance Amdahl product support. 
maemegmpany's SOfsWars deveslopmec: and programa enhance- 
MERtS ersure coMpatibility of its hardware products <*t0 tha 
Resemwedely Used Ssystens, end oths: software products are 


emedsat increasing ptoductivity of the user. 


Ae 





me SOLtTWare depact ment 25 2 part of the en 

G@ivision 2% Amdahl. The software quality assurance group is 
gmgemce OL that department and it sonsists of fiv - 
(Figure 4. 15) The main purpose of software int 

Remergmes fOr architectural intesrtace of its product with the 
customer's system. Because of this, the software develop- 
ment group does not have to start with any top down design 
Goeeesepzodguc. but £0 devalep complement software in order 
to tie the hardware products together. The. dea 


= 


for the development of software in this company i 


pl 
Ss 

Baeives hardware 9 its competitors, such as IBM. The 
authority of this organization d2penis on its c 

and expertise. The products that they release hav 


themselves in the market place. 


The quality assurance group of Amdahl's main inter- 
face with the development group cones at the end of che 
development cycle during the testing and measuring. The 
Peesomcanm= part in all tethnical reviews throughout <he new 
products development. The qualit assurancs group 2 
Micemw=ene program is "packaged correstly" for in 
This means shat «he software product aneets all the s 


of their competitor's systen. 


fOr new softwares about to be raleasad bv cthis 
company, “hey have what is known as <¢ a 

paogtam< The orogram 21ables the d 
Se@eewase an =o =h= field, test and debug it 


fo) 
Mmeeneae 2S =O DE applied before it is announced. 
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2. ‘Oliablity A 


Figure 4.4 shows the development cycle as it is 
perceived by Hewlet*+ Packard personnel. When the designers 
from research and development hav3 an idea for anew 
product, a proposal is sent to management. If permission is 
given, a product design group is forned consisting cf people 
from marketing, research and development, manufacturing, and 
guality assurance. When the design is laid out, the quality 
assurance people ask "Wnat aif" giastions +9 ensure all 
aspects are considered. Vee sCOMDahy:, SeiSons> particular 
specifications to which the designers must adhere, so they 


have the freedom *o be cre main languages used by 


ww 
(t+ 
{4 
<j 
(D 
6 
a | 
hv > Fi 
(Vv 


the designers are assenbler, Pascal, and FORTRAN because 
Their oroducts tend to be more techiical than rtommercial in 
nature. Pia Vere sompesqucc cuviconnental and applications 
software. One person from quality assurance is assigned +t9 
each project. 

During “*he requirement phase of the develo 
Mrgeess, an investigation has to be completed in order to 
produce a detailed specification plan and a user interfaces 
Sp=citication plan. In the external design segment «he 
quality assurance people must produse a qualit plan deli- 
Peatng the quality goals S9F objects of the project and ho 
they are to be measured. This is 2 problem area for «h2 
quality assurance people bsacause if the product is generaced 
@e a Customer's request, the request is usually not specific 
Oz incomplete. Beco ee Os aat es ena) sO=nelized conmunica- 
aliminate <2 


tions be established < Ps Problem. 


In ‘the intern of *he developmen 


fu O 
t~ 
th fa 
D 
— 
t* 
(t «Q 
S| 
se) 
po a 
jit 
(h} 
iD 


cycle, «he internal speci oles SOU RGOWO deSt¢5, and 
submodule design take ola v 
versonnel set wp, monit 


e 
Sime Geperet=* pets in d= 
and code reviews held du V 


SMC ecinic moss ods. The 


tie mune. Onal test plan. 
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PROPOSAL OR CONCEPT 


nee ee — enn eee oe. eee ee oe 


REQULREMENTS 
l. Investigation 


2. External Desiyn 


DEVELOPMENT 
l. Internal Design 
Implementation 
a.) eet retton « fest 
A. Functional 


B. Systems 


oes 


Quality Certification (Customer Acceptance) 
eo DoOdueehOnwCcehett: 1 Catton 


6. Manufacturing 
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UPERATEON 
Pigure 4.4% Hewlett Packard Software Developnent Cycle 
Seurce: Interview 326 Hewlett Packard Software Qualit Assurance 


Personnel 
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During the implementation statement, the quality 
assurance people set up the systens *est plan. Resiaal 
Zeeenegets NOt accomplished until the integration and +es*t 
seyjments, and it is done sn the function and systems levels. 
Meenetgn “he functional t2st plan is produced by the quality 

. 


d 
assurance people, mis Saetusl testing = is dsre by ths 


ih 


designers themselves. fas leved- 5 testing is viewed 
Mainly as a debugging exercise and would be a waste of «he 
quality assurance organization's tims and resources if dons 
by them. At the systems level of testing, the plan and 
tests are done by the quality assurance peopls. These tests 
Seem veewsd aS a thitd party auditirg inspeztion of «he 
BEOdUCT « This *hird pasty testing is done because Hewlett 
Packard does not believe that th program designers and 
analysts can be completely objective about their product. 
Mie se quali~y assurance group is also responsible for «he 
Beekaging of all test plaas for reusability. There are no 
percentages of correctness sought 

levels. When this segment is co 
Semsade-ed 109% correct.. 

NMeCOramaget@ ths jwalitty assurance people, ancther 
problem area is the schedulé planning. The designers do not 
eretmechae DrOoOblems will occur during this testing phase, so 
*hey have to be careful ts plan for 2xtra ‘time if problems 
CES Ur. 

Mewcree ne  CGieality certific 
basically a customer acceptance ins 
mee TeeteLeication —- ae 
Dicestge<enhaS .S2gment a oils run 2 
ensure chat 5f a customer requested the preduct, all the 
Peeeeats == Ene product itself, us2f manuals and 2ny other 


items -- are shipped. 


Wh 
WO 





Hewlett Packard believes in Weradl= | <G = Greve" 
involvement with its software products, which neans «hey do 


not abandon their customers after sala. Ad Hewiet. Packar 


f'« 


software is copyrighted so if there are any problems after 


Mersin Operation, th2 sco the customer is $100 per 
heuz for repairs unless the custoner has a subscrip«ion 
femevice.  Sudscription 


ed at. lower fee, 


h 
1 

service entitles the customer to hava 
sofiware repaired, updated, Eales 
h 


Cc 
This service includes a4 plan by which, if a program is 
updated or fixed for any customer, the updated version is 
sent out *o all other customers whe have «he same progran. 
Mae WeeisiOn =o use Zt within the customer's system is aft 
so the customer. 

Tf there is a problen, the customer first notifies 
the field activity which, if necessary, creates a “work 


around" program ‘to keep the customer's system operational. 


Peon the fpeld activity, Eno oaooleml ) 1S 2efsEped to the 
Manufacturer, via support, and eventually to the people in 
research and development who design the progran. They 
pmeeOme 7] =he problem ani place it in their schedule, and 
jm omeventually fixed. WOmnStaancale fCCOlds Of D=oblens 


or changes *o programs ar2> maintains. 


The quality 2ssutrance organization keeps abreast of 


Mieeiactes=: ideas and chanj2s in this field and is censtantly 
Seeevetg ©cO 2Mprove its ord oqren. 

Reference 

Personal interview with “ec. Rew mod Li Spear, software 
Produces assurance menager, 37 the Hewlett Packard plant, 
PosomoyecsoasS DevV-S2On, Cupertino, california, on 14 April 
198 2. 





Be TRW 


TRW is a diversified nultinational company which manu- 
factures a wide range of products from compon2nts for cars 
ani trucks +9 defense 2lactronics and space systems. TRW 
produces transistors, resistors, cap2acitors, disdes, poten- 
tiometers, trimmers, tuning devices, TV convergence yokes, 
SONPeECtOrIS, transformers, Betieedcicclarebecards, electric 
motors, electric data prosessing terminals, and jet engines 
Barts. Ocxther products include pumps, fluid handling equip- 
mnen*, nuclear reactor components, PeASen==sS, bearings, 
sieeeng ~DOl1S, and hand ts>1s. 

This company handles defense systems contracts which 
include the development of softwar2 and the construction of 


the entire system. 


lees aon iza= ion 


TRH is divided into many groups because of its 
G@eyers27=cation. One of thes2 groups, “he defense systems 
Gieme, contains the engin2ering divisicn of which the prod 
uctS assuranc? organization is a part. (Figure 4.5) Thes 
level is made up of managers whds are assigned to <*he 
G@ifferent projects in assistant prdject manager capacity. 


Mets departmen* is not ge concerned with software product 
assurance, but also wit rdware ani system engineering ani 
design (SEAD) product assurance. (Figure 4.6) 


Figure 4.7 shows the standard work breakdown struc- 
n 


dee for any product in ghe dafense systems group as it is 
concerned with product assurance. hie assus-ace. DTC jScz 
2 areas 


manager heads up a staff of personnel who work in th 
of quality assurance, configuration nanagement, relia 
amesele ty. 
Bngdrees. oO ShOv¥s he Sctandacd work bre2kqown struc- 
e Cc 


Mise =On the Guality assurance area o£ the 
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further subdivided into management, software, hardware and 
systen. 

When working on military contracts, the company must 
Bemmow SpeCiticazions required for sontract award. One of 
these specifications is MIL =S=52779A “Software Quality 
Assurance Program Requirements" of 1 August 1979. Laas 
document states the requirement for the establishment and 
implemertation of a software quality assurance progzan. i ie 
mBoenoped that this program could™b= tailored, 2conomically 
Panne a and developed in ‘eenjunction with the contractors 
programs of this type. PreecolcGact Or 2S Tequired to docu- 
ment this program in the form of a software quality assur- 
ance plan which meets its specifications. This plan has to 
miraentary Organizational raspensibility and authority for its 
execution and make timely provisions for special needs 
[feomerols, tools, facilities, skills, etc.). Because this 
mmodime Of the COmttact, it is considered to give the prod- 


ucts assurance organization its authority over the project. 


2. Nanagement and Softwares areas of the Projec: 


The standard duti2s expected ¢9 be perfor 
personnel in the management ar 9° 
POelows: (figure 4.9) 


a. Planning and Semtrol 


Gl\pen LO. OL CVseertetece- On Nand pacticip 
generaticn of qualit Soeteenee 21005 LhtoO the oro 
implementation plan, project scnedulss, documenta D 
ena Qehet Sinilar documents. 

(2) To define the qui 
lesen] APPEOpriate psrsennmel.s To a 


ance and prepare status reports. 





MANAGEMENT 


PLANNING 
AND CONTROL 


@ PROJECT PLANS 
e GA TASKS 
@ CONTRACT CHANGES 


QA PLANS AND 
PROCEDURES 


@ QA PLAN 
@e QA PROCEDURES 


PROJECT 
INTERFACES 


PROJECT MANAGEMENT 
PROJECT PA MANAGEMENT 
FUNCTIONAL PA MANAGEMENT 
TRW FUNCTIONAL MANAGEMENT 
PROJECT BOARDS 

QA OPERATIONS 
CONFIGURATION MANACEMENT 


CUSTOMER 
INTERFACES 


AFPRO 

FORMAL REVIEWS & AUDITS 
ASSOCIATE CONTRACTOR 
DOCUMENTATION REVIEW 


SUBCONTRACTOR/ 
SUPPLIER MGMT. | 


e SELECTION 
@ REQUIREMENTS DEFINITION 
@ MONITORING AND CONTROL 
FIGURE 4.9 TRW Quality Assurance Project WBS - Management Detail 


SOURCE: TRW Status Report on Standardization of Quality Assurance 
Functions Task, 20 April 1982 
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(3) Tormsonitor all actions in conjunction with 
contract and engineering changes. 


be Quality Assurance Plans and Procedures 


They are required to direct the generation of 
the quality assurance plan which follows the controlling 


i2w, maintain, 


<j 


government specification MIL-S-52779\4,to re 
migeipdae> It throughout =the project's life his planwas 
required to address: 

(1) execs - Peon aq= Ss, Methodologies and 
records to be employed in the performance of the work to 
Support the quality assurance objectives. 

(2) Procedur2s by which design documentation is 
reviewed to evaluate design logic, fulfillment of require- 
ments, completeness, and compliznce with specified 


standards. 


( 3) COnteaceer.s procedures for formally 
ep -ove.g OF certifying the Se 2M as nOotazat ton and 
completicn of wotk performed under contract. 

(4) Documentation of standar 


anda 

Sonvemc ons and practices to be used for a 

(5) DecuMe net euon y GF chs. .c 

meecomand CORtrolS fon handling o£ Source code and object 
code and related data in their various forms and versions. 

(6) DOCURS Beat. cn OL COP=ractor’s procedures 

for preparation and exacuticn of reviews and audits neces- 

sSacy in astablishing traceability ae Sal Tce hes ace 


requirements. 


Ger Vero ec. Lace ss.aces 


The management detail aldresses the interfaces 
between vcoroject manager, assistant project manager, subd 
Pa@mieee Manegsrs and Gthsrs 2r Conjuistion with che oroject. 
Reeweediae = 0 tne stafit meetings and c2spond =o action items. 





Gis 


Customer Interfaces 


The management detail works with the customer 
representative offices, MhOStSuben= tn) Visies Sand formal 
reviews and take care of documentation to and from the 
customer. 

2. Subcontractor and Supplier Management 

Pigure 4.10 delinates the dita s Oe «he 

personnel in the softwares area of the project. The +hres 


groupings are: 


(1) 


(2) 


(3) 


Management Support -- carries dsut duties 

in support of the management section of 

eRe prs joc 2. 

Engineering 

(a) Identify and define *the quality 

Standards and procedures that will 

be Followed ducing the design, 

development, programming, tes*inga 
and documentation stages. 

(bD) Identify software tools and special 

methodelogies that would be used in 

performance of quality assurance 

Rast 


use and ensure their us jiuring 


ct 
re 
iD 
| 
t4 


Establish procedures for 


ct 
a 
D 


DEQ) eC:. 


Pace pa ten Ge def 1s tson and Lnpienen= 


(Cc) 
Eee Orewa sonewa5e DLODLem Teper ting, 


Aner SiSpecOnt=ce2O) and Scon=tol systen. 


(Gj PAz®owe2pate in fotmal peviews, project 
boards and customer boards. 
(¢) Maintain recoris and files of documsn- 


taco review ©£Of adherence «=o 


stanjiards. 


Ore atvons 


cn 
WO 





SOFTWARE 
MANAGEMENT 
SUPPORT 


PROJECT PLANNING AND CONTROL 
QA PLAN AND PROCEDURES 
PROJECT INTERFACES 


CUSTOMER INTERFACES 
SUBCONTRACTOR /SUPPLIER INTERFACES 



















S/W STANDARDS AND PROCEDURES 
S/W TOOLS AND METHODOLOGIES 
S/W PROBLEM CONTROL 

FORMAL REVIEWS 

PROJECT BOARDS 

RECORDS MAINTENANCE 
DOCUMENTATION REVIEW 


OPERATIONS 
® 


AUDITS 
e TESTS 
e INSPECTIONS 
e SITE SUPPORT 





FIGURE 4.10 TRW Quality Assurance Project WBS - Software Detai] 


SOURCE: TRW Status Report on Standardization of Quality Assurance 
Functions Task, 20 April 1982 
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(a) Perform audits on project activities. 

(b) Participate in each level of software 
testing as designed by «he quality 
assurance plan and perform surveillance 
activities. 

(c) Perform visual inspecticns of ail 
software produrts purchased with hard- 
ware from supplier. 

(dq) Perform quality 2ssurance function at 
each sit2 and remote site for testing. 

moecunendg any = decumentation audit, a discrepancy is found, 
she 


2, the 
X 


the discrepancy is documented and is taken te Ghose 


O 


responsible designer. tapes aeG2ssatnh amoun= of cin 
Mepoteids not cozrrected, the problez is taken to then 
evel in the project orjgaalzation. The problen will tra 
up the crganization until the discrepancy is corrected eé 
if it méans going outside the project's environment. 
ADDEOKi Mewery 2 tO 5.55") 0f the entire project's 
Bomee 2s Charged *o quality product assurance, but it is the 
opinion of the managers 9 quality assurance in the TRY 
company that the cost of quality assurance 1S z2ro. 
Once a proiuct has been accepted by the 
mer, with the signing of defeise form DD250 Material 
ection and Receiving Report, the legal obligation of TRW 
nded. If any problems arise after release, the customer 
pays to have more work dons. 
Reference 
Oneal Se ecrview with “r. aiean vos BUCK, Produces 
pee Managems Mo. Samuel E. Benesch, Department Manager 
Mee aAsSSuranees and Ur. Manpean F. Kenchan, Senzo> Sta£tt 
Engineer of the Defense and Spac2 Sroup of IRW, Redends 


Beech, Calafornia on 7SHay 1982. 
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1. OLganization 


Figure 4.11 shows the strustar2e of the IBM organiza- 
e™on as of March 1982. I* shows that, under the staff 
level, the company is divided into two najor areas, 
marketing and service and manufacturing and development. 
Unier these areas, the Jrouping of divisions start in which, 
under «he informaticn systems anil technology group, =he 
general products division axists. 

The general products divisioi, with its headquar“ers 
lecated in San Jose, California, is responsible for the 
development of all hardware and software products at IBM. 
It has two development laboratories, one located in Santa 
mepesa, California a@nd@h>= other in Pucson, Arizona. (Figure 
4.12) 

The general products division is headed by a presi- 
dent with a vice-president in charjy= of each operational 


department including: hardware, software, manufacturing, 
Financing, support and products assurance. Heading ¢ach 


development laboratory is a center manager with functional 


Managers in charge of 2ach departmint below hin. Nee a 
each of the development centers, a functionai manager in 


Srasqe Of DroSducts Or quality assuraice. 


a 
The guality assurance déepartnent within this orga 
zation is completely independent of sthaer deparia 
software products déevelop2i in these laboratoriées 
Bro wenv-roOnmen® SEweperactional «ool area (Figure : 
they are produced in all of the major programming lan 
The quality assurance groap does have authority oo 

ucts that are néw and azz about +) be announced and over 
Peoeue=s that are beang shipped to customers. Ii chis grous 

agr 


dices  EO- Peo UMC emeeOE OLUGCs 2s ceaiy, 2% is sot released. 
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FIGURE 4.11 IBM Organization 
SOURCE: Interview with IBM Products Assurance Personne] 
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Figure 4.12 IBM Seneral Products Division 


Source: Interview with IBM Products Assurance Personnel 
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Figure 4.13 IBM Software Area of Development 
Source: Interview with IBM Products Assurance Personnel 
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The decision for product celease is not driven by any other 
mac tOrs. 

The quality assurance department is divided into 
three divisions, ‘two of which are oroducts assurance, and 
the other is verification and testing. Ever software 
product developed is diviied between the two product assur- 
ance divisions. The number of peopl= assiqned is a function 
of the project's size and their schedule depends or «hat of 
the developers. At the end of the develcpment cycle, all 

c 


BPeegimecs gO through the verification and testing division. 


2. Quality ASSuranse and Design Interface 


The qualitv assurance group interfaces with the 
program developers throughout the entire development cycle. 
(Figure 4,14) The peopl2 within this group have no prere- 


quisite skill reguiremant and most aave varied backgrounds 
ranging from programming 2x pertise to marketing skills. ie. 
do their job, they depend nainly on their experience and gut 
fealings. I: is not considered necessary for them to have 4 
programming or computer esaginesring background because? it is 
very tare that they have to inspect the actual cod=>= itself. 
Within each development dispartme 
who ¢xamine the code and t23st 
develooment cycle. 

The managers of cha development groups isvend on the 
Beople from products assurance for their objectivity and do 
not view «hem 2S a resource tcol. Phe SeuPEOGUS=S SSsuracce 


Dedple con<tzributSe to the oroduct in the following w3ys: 
eau Slannang 


Befors any work can be ettarted, a project plan 
Recmeeommce cU=) <Oget her in whieh th® pregramaers have <9 
aa Dos 


Slaps Sasso wai 


Ciaim which development style out 
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be used for this project. [This plan is named the 
Comprehensive Evaluation Plan (CEP) which also t2kes in 

account the quality assurance procedires, use of resources, 
and the project's schedules. It is considered the main plan- 
hing document and has to b2 approvei by the products 2ssur- 


ance divisicn before the project is started. 
bo Early Warnings 


If at any tim= during the developmant cycle, «he 
quality assurance inspector sees anything which might kee 
ths program development group from k2eping schedule, «hey 


Becefy zhe project manager. 
ce. Value Added 


Piye cure hgecono: DEOcess, the quality assurance 
people feel that something could be added +o the sof*ware to 


enhance or improve it, they inform “h2 development group. 
aq.  sduiciataon 


The education of the programmers on possible 
development tools, whether develop2ia in house or externally, 
memecastsed out by this »tzyanization. 

IBM sets standards requicements chat have to 
Beeereeco <Ne products, bus =here is flexibility in their 
Meemeoeeaise 1+ is ler® Eo the discretion of the programme2:. 

mae Vertflceeeoneeand +=se2ng veople carry out 
eee -unc-aonal testing at the =nd of the developmenc 
Pesiecss, perrorming besisally user ofiented t3s<s. Thee > 
MeeneeGbog=eG.oivye iS <0 debug thess products of any user 


Se=enzed problens. 


Besides the oroduct assurance, performance 
fasiieemard Ver rication and =2st groups interfaces, there is 
PeEeOaiee bua) =-2n A2v7ice for =nsuring quality products. 
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A Review and Inspection process (R5I) is carried cut by the 
programmers zhemselves throughout th2 development cycle. It 
is carried out either in a formal manner in which a meeting 
ls held with the programmers and a moderator and «hey 
discuss the program and its progress in depth, or it can bs 
held on arn informal basis with only the progr2immers! imme- 
diate peers present. A representative of the product assur- 


ance division is required to attend these meetings. 


cae Opesat ons 


Cnce a product has been released, the field engi- 
neering division is responsible for remedying any problems 
experienced by the customers in use of the product. This 
division is also responsible for maintainin Anh eS HO 2eal 
*racking record on probléens with the software products once 
in the field. If a projiuct is to be renewed or enhanced, 
mie products assurance people can cequest this historical 
information, but they are not required to keep track o 

T£ a completely new ovprodurt is released ob 
company for which the users would requir 1 g 
meeonsi bility for this tcaining is assumed by «he marketing 
dew asilon. Requests for new products a e) ie 
directly by the developmen+ laboratories, but thro 
=wOoumazn LBM user groups, SHARE and SUIDE, whith meet twice 
yearly to discuss problems and possible ideas for new prod- 

homie cting division is also constaaitly carrying 
Sue sieveys 2£ customers for new product ideas. 

he people of the quality assutancs department 
Bromcneecsac =+hei> main objective was tc maintain a wide 
range perspective of +h2 product ievelopment process and 


reyver to become overly involved with details. 


us 





Reference 


Personai interview with Mr. Baer oo eli. MeO adlande oa. 
Norman Towns or = he products assurance omen blo) = IBM 
Development Center, Saneaefebesa, cCalifornaa on 21 April 
MoS 2 « 

DIA AMDAHL 


Amdahl is a high <+t2chnolegqy company engaged in <=he 
eae -OL=tne-art design, develsonent, Me Mut ae aiceean a), 
Marketing and maintenance of large mainframe computers, 
software and communication systems. These preducts are used 
by large computer users in the full speéctrum of commercial 
@ed Scientific data processing environments. 

The company's central processing unit's design strategy 
26 to focus on the developaent of efficient design architec- 
—wmemrtO> high oerformance, @ependapiltey, aad flexibiizet 
BeEsEUCUre enhancement of the product 

The company's communication systems division designs and 
Manufactures — conmunication networks whic allow 
iMeens) tO interface wi multiple gzographically dispersed 
sys tems. 

Amdahi also offers Qmemumbes of Services O° Secs 
customers. There are programs for cros 
with specialists in both hardware ani softwa s 
There are also expanded 2ducational offerings with *ailored 
Pea ng -O enhance Amdahl product supoort. 

The company's software dev2lopmen*+ and prog 
Ments ensurs compatibility of its hardware products 
Nerscmepsely Used Systens, ahd Othe: software products ars 


aimed at increasing productivity of the user. 





ve Oxtdaneiza> ion 


The software department is 2 part of the engineering 
gey2ston at Amdahi. The software guality assurance group is 
apart of that department and it consists of five people. 
(Figure 4. 15) The main ourpose of software in the Amdahl 
Meeeedens £LO> architectural interface of its product with the 
customer's systen. Because of this, the scftware develop- 
ment group does not have to start with any top down design 
of its product but to dsvelop complement software in order 
mem-ete the hardware products togethsr. Die watt tye g LoOmce 
for the develonoment of software in this company is the inno- 
Werave Rardwate of <ts competicors, such as IBM. The 
@aenors-y Of this Ofganization depenis on its credibility 
eed espero ise. The products that they releas= have proven 


themselves in the market place. 
2. Development Interface 


The quality assarance group of Amdahl's ma 1g 
face with che development group cones at the end of the 
development cycle during the testing and measuring. The 


also take vart in all technical raviews throughout the new 
n 


products development. Tis GQualitcy Assurance group insures 

Biia=stne program is “packaged corrastly™" for installation. 

This means shat the software product meets all the standards 
@t their competitor's system. 

3. Operations 

For new softwares about to be released by cthis 

company, eevee Wwaat ms Known 2Se7the carly supoor: 

velopers to take “«ne 


program. The program erables the ds 
Eeteware into +t+h= field, te b 
b 


O 
were 2S =O be applied befcrS 12 2:35 announced. 
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After installation, if thers are problems with the 


software, «he field units of Amdahl handle then. There is 
the Amdahl warning svstem and maintenance tape, which is 
Malntained by the field units and, if there is a major 


problem, the software is sant back to the development center 
for rework. 

ieee adi nang tseeastlted ous For the Amdahl products, 
but there is a tramendsus in-hous? training effort on 
competitors! equipment. 
Reference 


ime2eer View With Mr. Richard L. Patrick, Manager, Softwars 


Quality Assurance Group at Amdahl's. 
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This chapter gives the reader an analysis and summary of 
the interviews with the commercial computer conpanies and 2 
comparison with the FMSO environment. At 7 -ne enaee of gh= 


chapter, conclusions and tecommend2tions are given. 


me «=OUESTIONS FROM INTERVIEW: 


hi Where does the quality assucance group fit into ths 
company's organization? 
a. Hewlett Packard 
The products assurance jroup is a part of «the 
data systems division and is on th2 same level as engi- 
neering, manufacturing, marketing, and other departments of 
eps G2iViSion. The products assuraace group fits into che 
@e@ipany'sS otganization in a line function position. 
De LW 


The products assurance group is a part of th 


(Dp 


engineering division. Die seeeoundecs ts ano the company's 
meapanszet On in a staff function. 
Cae 2 ei 
The products assurance jepart 
the software development center. ee 
Sane level as the development departnent o 
ener Luance On. 
ad Amdahl 
The software Juailty assweance group is a part 
cif the softwate dé=partnent. ieese besa tioned on the sane 
level as the research and development g 
u 


G@iieeety asSutanc]S group is in 2 line £ 
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Ce FMSO 
Miemiiatean ye CONntsOlL Nbanen exasts in the manage- 
fememoepal=etent, Code 92. Tt is in 2 staff function. 
2 What type of authority/power does the guality assur- 
ance group have over their software product? 
a. Hewlett Packard 
This group's power relies on its a 
persuade management that the product is not ready and its 
meme at ON. 
ie TRY 
THe eauhority of this quality assurance group is 
given by a contractual requirsment, MIL“-S-S2779A "Software 


Quality Assurance Program Requirements." 


Gc. St 
The products assurancS group has completes 
authority over software product. Hae on LS Gsoun Eselsi st hae 


the product is not ready, it is not released. 
ad. Amdahl 
The software quality assurance group's power 


Memes nc PZOdUct depends on th2= group's credibility and 


€. FMSO 
The quality control group ¢xercises administra- 
tive power cver products. It insures that the quality 
Pomme eucheck—ort list 13 pzoperly €illec out and that “the 
product meets sp2cifications. 
Bre Whee weatidigefscatz=ons do =he  peoole im ths quality 
assurance group have? 
@e Hewlett Packari 
Themwe iquality. assira 
~omuames 2notdh education sand experience to 53 pregra 
and designers. 
Dome (Lan 





Ge Won 
Nepspectfie qualification required. 


d. Amdahl 
New Speecitac qua lificatwon required: 
2. eter LO 


PerSomnpet in the quality control branch are 
expected to have a complet2 knowledge of the system develop- 
ment process, from all aspects. 

Hee Row does the guality assuraice group interface with 
the design/development group? 
ae Hewlett Packard 

The quality assurance personnel are a part of 
fis product development group and work with the product 
designers throughout the developmant cycle. They are 
required to produce a quality assurance plan which states 
the measurements of the quality objectives and +o partici- 
Meee an the product testing on both the functional and 
system levels. 

a ine 

An assistant project manager 1s assigned to 
Syesyo-ojec*, with his own staff, to coordinate and partic- 
Meeetenear the quaizty csatrol funstions required in the 
Pao ject. They pecrforn 20d5—) testing of the product and 
Bact crpate in all technical reviews. 

Ge BU 

The product assurance peopl meme er rac 
software development personnel he eeast css the @ 
Gyo Le. They aporove the program deveslooment ola 
Mameagenenc <initormed of anything that might affect the 
Baagre='sS schedule. iiey §COrnOe. Pests cl1pacs inh 1D 


Ronee = sete ate =wo third pasty groups, the perforn- 
1 


v3 


Becomesoup and ihe verifica=2on ani t¢st person 


Gaee 7 Out vahis functgon. 





qd. Amdahl 
The software quality asarance group inter 
with the development personnel at the testing and o 
ment end of the development cycles. They insure th 
product is "packaged correctly" before release. They ares 
required to attend and participate in all technical rev 
G@urang the development of the product. 
€e&. FMSO 
PhewdWall cy seontrol, peanch checks the functicral 
description and systen specifications administratively. 
[Mievee2nsure that the quality control check-sff list is 


filled out properly and participats in product testing on 


ay 


very infrequent basis. 

As shown in the question, all of those interviewed, 
axcept TRW and FMSO, hai their software quality assurance 
@eogesean 2 laine function position in the organization. bles 
should be noted «hat the products assurance group of TRW was 
in charge of a line management staff which was assigned to 
PeemepEocuc- =O perform in a line function. Pies) faere 


momoniy the staff group. 


eee OPS nh Onur ore ene autnor Of this thesis <+hat 
gGuzstions 2, 3, and 4 tie in together. In all the companies 
interviewed, the quality assurance group is considered and 


functions as an integral part 2f the development tean 
work with the development personnal throughout the develcp- 


ment cycle, relieving any advisary situation. 


If the personnel in the guality assurance group do not 
Mewemene expe=.1Se =O Sarzy our Zesting of <=h2 praduct, 32 
eieszae Dare y in th opment 


Gumpec onbany SS SeGaniza tion dc. Dev 
le) 


el] 
personnel cannot be expessted to > completeslv objective 
D m 


Soe athe: r OwWh product => perEorm its *esting. 
Because the quality assurance personnel work alongsids 
thea development peopl and perforn som forn Of audit 
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mUnGt LON , 


enema Opin on has ecredib:li 


people and management. 


This has a direct 


authority over the product 


eho O » 
gntegral part 


any 
people in 


aos uwk ance 


aid & ing 


Se 


the 


Shengqualsiy “sontrol 
the development 

PUunGe 10 7 
GDAS GCakbay Out all tes 
eneck=of fist 


quality control branch has 


#ean. 


on +he 


is 


ping the product's release. 


5) - 


ae 


that were 


Ce 


we > 


ct 
2 Be 
yy 


cr 


aS e 
we be Ss 


What *ools, 


guality aSsurance group us? to do their 


methodologies, 


Hewlett Packard 


Nos tools; 
unique 
TRW 

No. tools, 
unique 
IBM 


No -caiols, 


Wheaue to fhe qui: 


Andahl 


NiO COOLS, 


methodologies 


Peowuc =: 
re ch jelge pe 


ly 
no real justi 


complete 


2 


2 


ef Eee. 20n 


They rarely pe 
The develo 

fee he gu 

led out 


1 3) 


fai 


e. Cave EO] 


cechnigues doe 
5 ON Obey 


techniques were 


to the guality assurance function. 


methodologies or techniques were 


methodologies 9 


“y assuran 


to the quality assurance function. 


methodologies or techniques were 


ipudije weOme Te qialicy d=sutance furction. 


FMSO 
Nowtoo ls; 
unigue to 
question, 

The 


*ools 


they used anything 


programmers 


nethodclogies or techniques were 


he 
— — 


unigu2 


Hone: OF 
gk ade 
techniques 
which, 


and 


hality of =the sotgware 
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ty with the developmen: 
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branch does not become an 


crform 
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used 
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programmers write better programs. These tools and *echni- 
ques were acquired through the survey of computer science 
literature or developed within th2 company and passed on. 
No compan interviewed wis willing to share any of these 
Meoloewern the author of this thesis because their tools 
were of a proprietary nature. 
There are companies that devalop tools and provide 
services which aid in the areas of programming and quality 
assurance. One such company is Software Research Associates 
(SRA), headquartered in San Francisco, California. A 
description of the purpos2 of this comoany and its activi- 
#i3S5 32S provided in Appendix B. 
aye Are historical records kept of problems with soft- 
ware products after their release and who in the company's 
organization keeps them? 
ae Hewlett Packard 
No records of this type ar2 being kept at «his 
time. 
Db. TRA 
No records ar2a kept of product problems after 
release. 
ce eit 
Historical resords of problems ars kept by «he 
field engineering division. 
d. Amdahl 
A maintenance ‘tape of problems is kept by the 
mold Engineering division. 
eC. FMSO 
Records are maintained dy the qu 
bomen chaough analysis of Frogram Troubls Reports (PTR). 
ic Who handles problems with software after release, 


ani how ere such problems handled? 
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ae Hewlet* Packari 
Problems are handled by Field engineering activ- 
ities who build “work arounds" for tustomers if necessary. 
If there is a critical problem, the software is returned to 
the development group for repair. 
be TRW 
TheLemesmnoOmlagalsODl liga. On On “h= part of ths 


company to handle problems after a product's release. ie 


$v 


customer desires TRW to fix a problen after product release, 
the customer will be charged for ths services. 
ec Len 
All problems are completely handled by the field 
engineering division. The software is not returned to «hea 
PTabOocatOryY, Nowm=attier how critical. 
d. Amdahl 
Droblems are handled by the field engineering 
Gad Up. Pf enere =2s Vasnajor™ problen, the software is 
returned to the development personnel. 
e. FSO 
The software is reported to tne CDA and 
repaired. 
a6 If a brand new product is desigred, who in thea 
@em#many'S Organiza~ion irains the customer on this product? 
@.e Hewlett Fackari 
Maaco = egda ves  Opmearriss OUT. th aIning. 
So GaRW 


NO Gsaeieeng 25 Cazzte@m@ema: by the company after 


Cen ube 
Moukecindg division caimritss out training. 
d. Andahl 


Memk= = foe diy se On calcaes Out trayvaing. 
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e. FMSO 
Pic timeccan ns Munl<ts go t> activaties from the 
SDA S. 

A question that might have been asked jiuring these 
interviews concerned the effectiveness of the company's 
software quality assuranc= program. The author did not ask 
this question because it would be improbabl2= to expect an 
objective answer. Thais thesis did no+ offer a quantitative 
measure of these grcups’ performances +o make its compari- 
sons. The euthor's intent was to tompare their view of the 
quality assurance organizatior's rol= and how ¢hey function. 


Baw SUMMARY CONCLUSIONS 


The purpose of this thesis was to investigate the 


methods used by large connercial conputer companies in ths 


4 


area of software quality assurance. The primary objective 
was to see if any of thes practices could be used in FMSO's 
environment. 

ie The greatest difference between the commercial 
companies and the PMSO environment was in management's view 
ance group shculd 


Semewnas Lole or function a quality issur 
the trend of thought 


fake. In the commercial 2nvironment, 
is that the quality assurance rol2 is a line function that 
Comrade pe GOntrolled from a Staff position. fa fus@ yen 
quality assurance role is only being fulfilled through a 
Seace DOS tt On. 

2. There was a difference in the 
ance personnel interfaced with 
the commercial companies ened 
became an integral part of the de 

v 


fons ama actions bez 


‘rj 
He 4 
" 
QO 
= 


Ng 
project managers. AE al 
Pow Sent DOSiaion, da 


o2 
Temes stfam, =~1nuS creating 22 ad 





C. RECOMMENDATIONS 


ue hooters Colalon Of  <<h= atthor that -mmsOo should 
emenge the quality control branch's ososition from a staff to 
ae me fLunction. As shown by tha ilaterviews, this is ths 
Peena Of thougnt on the position of an organization of this 
type in a software production 2nviroanent of toilay. 
ae In the PMSO environmant, to convert the quality control 
Meee’ S POSltion from 2 staff t> a line function, an 
increase in the branch's size would b2 necessary. 

This could b2 accomplished in ‘t4#9 ways. Ine way would 
be to hire more people tos increase its size. The other 
Manner would be t+o0 take people already in the CDAs and 
assign «hem the specific job of quality assurance. The 
second manner may be aors effective because these peopls 
would already be acclimatei to the P4Y5) environnent and have 
the knowledge of practices in theic own CDA. People of 
experience and expertis= could be chosen and, since already 
known by the personnel in their davslopment groups, would 
not be viewed as outsiiers. They would be able to either 
@aeay cu Or be in charge of the audjilting funstions in «he 
software development process. FMSD gould not have +o change 
its development precess. DiS att weUnCcuen. Of DOSLELON 
eembaest3l1i be held in Coil 92, but it would be in charge of 
a lime quality assurance organization in che CDAs. 

B. The Qualty Assurance Chacklist could be used as “ths 
quality assurance group's work description document. They 
Melied be in charge of s2artying out the saleanents of the 
@eeoerst in a thicd pacty ~auditiag function. Because “ha 
checklist points out ‘the segments jdluring the developmen 
Peorcess ewhere surveillencs for guality is important and tae 
list covers the a2ntire development process, it would be a 


very useful guidelines. 





Looking at the first element of the checklist, tha scope 
of releas¢, a separate chacklist should be made up for each 
Semenue four levels of projects to cut down on confusion of 
which elements should be jone for which project. 

The elements stated in the chacklist are also very 
booed. Melo be SPCC EITC I> Scription of tthe tasks that yould 
have to be carried sut by the guality assurance personnel 
faa be promulgated. This description of tasks would also 
have to coinsidé with tha steps of the system development 
peace so. 

The quality assurance staff function in Coie 92 should 
monitor the projects progr2ss and b2 involved in it's DPOAEM 
ease. They should have Final authority over the this nila- 
stone plan. They should attend all oroject internal reviews 

Mompascapat> in, if no more thar monitor, all testina. 
G4. With the quality control branch in its present positicn, 
Zt as the opinion of the author that 14 1s a waste of this 
Segenmeazat.on'’s time and resources 9 be involved in the 
collection and analysis of Program fYrouble Reperts which 
eecord problems after software release. The only 


Srgenizget:on t5 which this type of information is important 


meeene Organizetion which developed it and has to fix it. 
Al S Seganizgation should 2xpend its energy in the 
Maintenance of these types of racords, and the quality 
assurance people should nonitor them. 

Se An effort should be made by FMS) +o maintain records of 


in-house development *ools that could be shared between che 

CDAS. Mae assastance of a tool davelooment orgeénazation, 

such as Software Research Associates, Gould Oe SOUugh=. 2s 

Memomterea 12 #jwhe aréas of program davelopment and sottwara 

Ginesuey con=rol tools. 

6. Maman S-2f£2cat:02 need be supplied f5r acquitina 
= 


resources <=9 ccomplish thes gaals, she requirement 


ef 





invoked on civilian contractors for a software quality 
@esuiance prEOgGram, MIL-S-32779A, souid be given. If the 
government requires this extensiv= a) program. “FOr “ics 


eveliaan CONtractors, why not requires it for itself? 





APPENDIX A 
FMSO SYSTEM DEVELOPMENT PROCESS 


2.3.2 System Development Process (SDP) 1s the function by which FMSO trans- 


forms a Requirements Statement into a documented, functioning set of computer 
programs and procedures. FMSOINTNOTE 5230 of 21 Nov 1979 established the CDA 
Development Process Model provided as Figure 2-4. The CDA Development Process 
Model reflects all of the basic steps appropriate in ensuring that each CDA 
Tasking received by FMSO is effectively managed and results in a high quality 
product being released for use by the customer. The model covers all projects, 
large and small, new development or maintenance. However, it is anticipated 
that some of the steps in the model may not be applicable to all projects. 
Therefore, an explicit decision by the appropriate level of management is 
required in order to exclude process steps determined not applicable on a 
project. 


2.3.2.1 Definitions of Figure 2-4 Symbols 
elope ee ae (Line Management Review and Approval). This responsibility is 


assigned to FMSO Department iine managers thact hat’: been tasked with the 
development of a Project or resolution of a Program Trouble Report (PTR). 


D3 2d "(Top Management Review (Optional)). This responsibility is 


assigned to a Project Review Board appointed by tne Commanding Officer to 
review designated Command-interest projects. The Commanding Officer will be 
final approval authority on these projects. 


2.3.2.1.3 "}" (Management Department (Code 92) Project Tracking). This 


responsibility is assigned to the Management Department to administratively 
act as FMSO's front door on all Project and PTR tasking, and to track progress 
for the Command via the standard FMSO project status tracking reporting system 
of specific Command-designated projects. 


ks Oe ie We Pa BT! (Management Department (Code 92) Project Management). This 


responsibility 1s assigned to Code 32 for projects that have significant 
Critical interfaces in two or more Departments for which the Command has not 
specifically designated a Project Manager. Project Managers will be the 
Command focal point for the project and provide the coordination necessary [9 
ensure that all significant/critical interfaces are resolved. 


2.3.2.1.5 "@" (Management Department (Code 92) Quality Control (Q/C)). 
This responsibility 1s assigned to the !anagement Department to assure that 
all line management tasking has been achieved within FMSO Q/A standards. 
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peo. 1.6 oO" {Management Department (Code 92) Quality Control (Q/C 


Optional)). This responsibility is assigned the Management Department to 
perform selectively at their discretion on designated development process 
events. 


fone. 7 YO" (Manacement Department (Code 92) Line Management). This 


responsibility 1s assigned to the Management Department to perform line 
management functions for designated development process events for all 
projects where applicable. 


2.3.2.2 Descriptions of SDP Model Steps 
2.3.2.2.1 Tasking Requirements Statement (RS) or Project. The development of 


a Requirements Statement (formerly entitled the Systems Policy and Concepts 
Statement) is the responsibility of the system proponent; however, current 
Command policy is to provide assistance in the preparation of the RS by the 
system proponent (where warranted and approved by the appropriate Department 
Director or Project Manager). The RS or project tasking document will be 
logged in by Code 92 as a Project Tracking function and forwarded to the 
responsible department(s) for acceptance or rejection. 


2.3.2.2.2 System Definition Acceptable (SYSDEF OK). Line management will 


review the tasking document to ensure that it contains sufficient information 
from which to develop a functional description, cost benefit analysis, plan of 
action and milestones (POA&) (internal or external), resource estimates, and 
priority acceptability. If sufficient information is not provided, a letter 
citing tasking deficiencies will be sent by line management or by the Project 
Manager (if appropriate) to NAYSUP with a copy to Code 92 to stop Project 
Tracking. Tasking must contain the general definition or the target hardware/ 
software environment to be used or it must be clear that an existing suite of 
hardware/software is intended. When tasking is acceptable and the project is 
a new development, is a new Application/Operation, changes disk files or 
teleprocessing, 1s estimated to exceed 1,000 manhours or FMSO etfort, or 

may impact system software, a copy of the project will be sent to Code 94 

to provide estimated costs or determine that system suftware is not affected. 
Code 94 will respond to application Departments within two working days in 
either case. When tasking is acceptable from ail of the above. line manage- 
ment will return a copy ot the project to Code 92, with total estimated costs 
annotated, for a Cost Benefit Analysis. 


2.3.2.2.3 Cost Benefit Analysis. Code 92 will develop a Cost Benefit Analysis 
with the assistance of line management. If not cost beneficial, Code 92 will 
prepare a letter to NAVSUP rejecting the project, update Project Tracking 
records, and advise line management and the Project Manager (if appropriate) 

to stop further effort. CBA may be subsequently iterated at the discretion of 
Code 92 or line management. 


2.3.2.2.4 Estimate Resources. Line management, including Code 94 if involved, 
will develop initial resource est:mates and determine priority acceptability/ 
required to perform the tasking. Resources include personnel, test bed and 
operational hardware, software, travel and overtime requirements. [f there is 
a shortfall, line management or the Project Manager (if appropriate) will 
prepare correspondence (including an impact statement) to NAVSUP requesting 
additional resources or a cnange in priority. A copy of the letter will bde 
forwarded to Code 92 for Project Tracking. 
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2.3.2.2.5 POA&M. Line management, including Code 94 if involved or the 
Project Manager (if appropriate), will develop internal and external POA&Ms 

for CO-designated projects as discussed in paragraph 4.1.5.4.2. Examples of 
POA&Ms are provided in Appendices 4.1-A-1 and 4.1-A-2. A copy of the POA&Ms 
will be retained by Code 92 for Project Tracking. The CBA, resource estimates, 
and (for CO-designated projects) POA& will normally be done concurrently and 
included in a letter to NAVSUP including a commitment date for FMSO to complete 
the Functional Description (FD). In addition, FMSO line management or the 
Project Manager (if appropriate) will update external POA&Ms monthly for 
submission to NAVSUP (NOTE: A senior executive Project Review Board (PRB) has 
been established to execute FMSOINTINST 5200.7B. Line management will, on 
Commanding Officer-desizgnated projects, provide or present to the PRB a System 
Definition Review in accordance with FMSOINTINST 5200.7B. When this is approved 
by the PRB and subsequently by the Commanding Officer, line management will 
prepare a letter for the Commanding Officer's signature to NAVSUP stating the 
official FMSO position). 


2.3.2.2.6 Approve POAGM. Code 92 will monitor this event as a project track- 
ing responsibility. when the approved POA& is received from NAVSUP, the next 
three steps (i.c¢., retime hardware requirements, provide ADS plan, provide 


resources) will be initiated concurrently. 


2.3.2.2.7 Retine Hardware Requirements. [f required, NAVSUP will refine the 
hardware cequirements at a level adequate for inclusion in an ADS plan. Code 
92 will monitor this event for progress as a Project Tracking task. 


2.3.2.2.8 Provide ADS Plan. If required, NAVSUP will develop or update aa 
ADS plan and process 1t up the chain of command for approval. Although it is 
recognized that further FMSO development of the tasking should wait for ADS 
plan approval, this nas proven to be impractical. 


2.3.2.2.9 Provide Resources. If required, NAVSUP will provide resources 
and/or priorities necessary to execute the POA&M. Code 92 will monitor this 
event for progress 3s 4 Project Tracking task. 


2.3.2.2.10 Develoo Functional Description (FD). Line management will develop 


the Functional Description (FD) and submit to NAVSUP for approval, including 
retined estimates ot rcesources per paragraph 2.3.2.2.7, above, with a copy to 
Code 92 for Project Tracking, Quality Control, and compliance with standards. 
Upon completion of the FD, line management or the Project Manager (if appro- 
priate) will conduct a System Design Review. On Commanding Officer-designated 
projects, the review wiil be provided or presented to the PRB in accordance 
with FMSOINTINST 5200.78. Code 92 will provide or present an updated CBA as 
appropriate. When aporoved by the PRB and subsequently by the Commanding 
Officer, line management or the Project Manager (if appropriate) will prepare 
a letter to NAVSUP, for Commanding Officer signature, including an updated 
POA&M with a commitment date for FMSO to complete the System Specifications 
(Sor 


2.3.2.2.11 Approve Functional Description. NAVSUP will review the FD and 


approve, approve «ith gualifications, or disapprove. This is the critical 
path to the development »f the System Specification. NAVSUP will update 
cesource requirements as required. Code 32 will monitor this event tor 
progress as a Project Tracking task, if required. 


ae 





2.3.2.2.12 Acquire Hardware. FMSO assists by estimating capacity needed for 
a representative site. NAVSUP coordinates with other NAVMAT or Fleet claimants, 


performs data call to all affected activities, and determines system-wide 
requirements. NAVSUP, directly or by notification to other claimants, initiates 
acquisition. Code 92 will monitor this event for progress as a Project Track- 
ing task, if required. 


2.3.°.2.13 Develop System Specifications (SS). Line management will develop 


the SS for release to customers with a copy to Code 92 for Project Tracking 

(if required), Quality Control, and standards review. In addition, at the 
completion of the SS, line management or the Project Manager (if appropriate) 
will, on CO-designated projects, provide or present to the PRB a Computer 
System Analysis Review in accordance with FMSOINTINST 5200.7B. In addition, 
Code 92 will provide or present an updated CBA if appropriate. When approved 
by the PRB and subsequently by the Commanding Officer, line management or the 
Project Manager (if appropriate) will prepare a letter to NAVSUP for Commanding 
Officer signature, including an updated POA&M with a commitment date for FMSOQ 
to make the proeram release. 


2.3.2.2.14 Provide Test Bed Hardware. NAVSUP provides hardware and system 
software (if any) needed for program development and testing. Code 92 will 
coordinate or arrange the installation. Since this is the critical path to 
process event 2.3.2.2.16, program development can begin but not be completed 

if test bed augmentations or acquisitions are needed but not provided. Code 

92 will monitor this process event on projects where test bed hardware/software 
is required as part of their Protect Tracking function. 


2.3.2.2.15 Program Trouble Report (PTR). PTRs will be received by Code 92, 


logged for PTR monitoring as part of their Project Tracking function, and 
forwarded to the responsible department for resolution. PTRs may affect any 
development process step in this model, and are discussed in detail in paragraph 
G2. 5. 


.-3.2.2.16 Program Development. Line management will develop Program Speci- 


fications (PSs), develop programs, perform unit testing, develop Program 
Maintenance Manuals (MMs), Users Manuals (UMs), and Computer Operation Manuals 
(CMs). PSs, UMs, and OMs will be released by line management to customers. 
Code 92 will provide administrative documentation release services including 
review of the documentation for completeness and compliance with documentation 
and system development process standards. 


2.3.2.2.17 Develop Implementation Plan. The customer is responsible for the 


formulation of a systematic implementation plan based upon individual customer 
requirements. However, FMSO must assist the customer on some projects by 
developing a proposed plan and negotiating the issuance of a plan by the 
customer. Negot1rations on the implementation plan will be performed by Code 
92 as a line management function for designated projects, with assistance and 
review/approval by line management in atiected departments. Impiementation 
plans required on projects not designated for Code 92 development will be 
developed by the appropriate department line management. 


2.3.2.2.18 Testing. Test Plans will be developed and string tests and/or 
system tests will be performed by line management. Code 92 will selectively 
review test plans and test requests for compiiance with Quality Assurance 
standards and procedures. 
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2.3.2.2.19 Provide Hardware to Field Activities. NAVSUP and other claimants 
will provide required hardware capacity, if any, for field activity implemen- 
tation. If required, Code 92 will monitor this event for progress as a 
Project Tracking function. 


2.3.2.2.20 Program Optimization. Line management is routinely responsible 

for program optimization. Code 92 will select programs for review and process- 
ing through available optimization tools, and provide any solutions developed 
to line management by formal memo with logic changes specified. Line manage- 
ment will schedule and modify the programs in accordance with the solution 
provided or resolve with Code 92. 


2.3.2.2.21 Independent Test Group. An independent test group will be estab- 


lished in Code 92. For Code 92-selected projects, entire release packages 

will be Quality Controlled for compliance with standards and procedures, clarity 
and ease of implementation. Aiso, all output products for the selected projects 
will be reviewed for quality. In instances where this effort will be accom- 
plished prior to program release, line management will be advised during 

initial POA&M development for inclusion in estimates. Recommendations for 
Changes or corrections will be made to line management. Line management will 
make the changes or corrections in accordance with the Code 92 recommendations 
or resolve with Code 32. 


2.3.2.2.22 Release Programs. Line management will release programs for 
Operational Review, Prototype or Implementation when all Q/A functions have 


been satisfied. When released for prototype, line management may withhold 

program releases to other customers for implementation pending successful 
prototype. Program Trouble Reports (PTRs) or Flash notification will normally 

be forwarded by a prototype activity to FMSO. Code 92 will provide administrative 
release services in accordance with current procedures, coordinating the release 
of environmental and application software and coordinating resolution ot hardware 
and software intertace requirements. In addition, Code 92 will review program 
releases for completeness, clarity and compliance with documentation and 

system development process standards as a Code 92 Q/C function. If required, 

Code 92 will monitor this event for Project Management or Project Tracking. 


Senco ce eo OBeReViewsOr Prototype. Thrserse the responsibility of the customer 


and the primary participating responsibility of line management. When this 
occurs, Code 92 will participate at their option as a Code 92 Q/C function. 
If required, Code 92 will nonitor this function for Project Tracking. 


2.3.2.2.24 Implementation. Implementation is a customer responsibility with 
support provided by FMSO. Support will be provided by Line management and/or 
Code 92 in accordance with the implementation plan. If required, Code 92 will 


selectively monitor this event for Prosect Tracking. 





2.3.2.2.25 Post Release Review. As a Quality Control function, post imple- 
mentation visits will be made to selected sites by Code 92, at their option, 

to determine whether the FMSO program release satisfied the tasking and whether 
the activity 1S using ‘t properly. Feedback will be provided to line management. 
An attempt will be nade to verify that the expected benefits were achieved. 
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APPENDIX 8 
SOFTWARE RESEARCH ASSOCIATES (SRA) 


Sofzware Research Associates 


ABOUT SOFTWARE RESEARCH... 


Software Research Associates (SRA) is an advanced technology research and engineerinz 
frm involved in software science, software engineering, software quality assurance, and 
software maintenance. The main activities of the Company are education, researzn aneé 
development, consulting, software tool design and production, and allied cechnizal 
services. The Company has offices in San Francisco (headquarters) and Los Angeles, 


California. 
Professional Development Technology Seminars... 


The Company offers series of Professional Development Seminars on a periodic basis 
publically, and on an in-house basis as well SRA seminars are distinguished bv their 
dedication to presentation of state~of—the-art software engineering technicues. Se miner 
offerings currently includes Software Quality Assurance, Applied Verification Tecnnisues. 
Advanced Software Validation Techniques, Automated Software Engineering Tooi 
Technology, and Software Maintenance Technology. 


Research and Development... 


Company researchers track the latest technical developments in a range 9 areas. 
including software production, software cesting, and software maintenance. as wil 2s 
otner zreas of software science and ensineering. Typical Company research protects 
have included work in such areas as: techniaues for validation of software engine2rn¢, 
systematic automation of the maintenance “Zunction, and general nethodciogies Zs 


comprenensive software testing and analvsis. 


Consuloing and Technical Services... 


eae 


Consulans ‘or Comsanv clients has range? from evaluation of advanced corputes 
archicecturas 9 che design of state-of-the-art software quality assurance crvanizations. 
The Company's approach to consulting emphasizes complete technical disclosure so tne: 
client organizations can make enlightened choices between technical alternatives. 74 
Compeéenvy alsa provides specializec technical services using advanced soztwar 
engineering tools. Such services include software quality assurance, software testing, 


and software maintenance support. 


Pubhcanions... 


The Company publishes a quarczerly newsietter, "Testing Techniques”, that is distributed 
wirhout cnarge to qualified technologists throughouc the world. The new newslertar, 
"Quahry Management Monthly", is focused on applying quality management technicues 
throusnout the software life cycie. The Company also publisnes ‘in orinted and 
machinabie form) the "Software Engineering Automated Tools Index“ tha: desczes 


some 500+ software support tools. 


Software Engineering Tools... 


The Comsany provises scftware production. testing and qualicy assurance. 


Maintenance cools for a variety of somouter systems. The S&TRAN syst2= 
structured programming preprocessing provides advanced confrol structures. ato mate: 
srogrem documentation, and auzeosacic instrumentation. The TCAT svstem for sciiwar 
system test coverage analysis orevides a guantitative dase icsr quality assuramce testiq 
of TO8GL rrocrems. The ITB interacstive software analysis 2anz testing i22uety 27 s.vs 
a¢vances analysis concepts for susport 2f interactive software qualcy asserarce. sac 
ISUS svstem for semantic uodate snd mainrenance 2f software systems c2creserts a7 

2 software sonfigur2tion stanteecent enc ord% sion 
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APPLIED VERIFICATION TECHNOLOGY SEMINAR 





Quality in a software system is a function of logical integrity of every part of the system 
and of the system as a whole. Verification (or "proor) techniques are used to help 
establish the needed levels of integrity. 


This new SRA Software Technology Seminar describes applications of the “proof of 
correctness" methods to software system quality control. In the correctness proving 
approach conjectures are formulated which express correctness with respect to 
specifications. The conjectures are generated by combining assertions 2bout the program 
oehavior with information from the program source text. These conjectures are then proved 
using information about the "meaning of the programming and specification languages, 
mathematical logic, algebraic manipulation, and mechanical theorem oproving. The 
methodology that surrounds the AFFIRM system will be described in detail 


This seminar is intended both for individuals in R&D positions and software engineering 


personnel working on highly reliable computing systems. A brief outline of the main topics 
in the seminar is: 


PHILOSOPHY AND MOTIVATION: What is Verification?; Programs as Mathematical 
Objects; Unification of Verification and Design. 


THEORETICAL FOUNDATIONS: Inductive Methods for Programs and Data; Proof 
Rules ‘tor Simple Control Structures; Axiomatic Specifications for Data 
structures; State Transition Systems; Foundations of the AFFIRM Approach; 
Styles of Mathematical Proofs. 


VERIFICATION METHODS: Inductive AsSertions; Recursive Functions And Their 
Proofs; Proofs of Data Structure Properties; State Transition Proofs. 


TECHNOLOGICAL SUPPORT: Verification Conjecture Generators; Formula 


Simoplifiers, Rewrite Rules; Interactive Mechanical Theorem Provers; The 
AFFIRM Approach. 


SURVEY OF APPLICATIONS OF VERIFICATION: Security Kernels; Distributed File 
Systems; Communication Protocols. 


The instructor for this seminar will be DR. SUSAN L. GERHART, Technical Director of 
Software Research Associates, Los Angeles, California, a post she has held since Octover 
1981. [n this capacity she is concerned primarily with the application of verification 
technology to practical problems of software and system quality engineering. 


Dr. Gerhart earned a B.A. from Ohio Wesleyan University, a M.S. from the University of 
Michigan, and a Ph.D. from Carnegie-Mellon University. After serving on the computer 
science {aculties of the University of Toronto in 1972-73 and Duke University from 1973 to 
1977, she joined the Program Verification Project at USC Information Sciences Institute. 
There she oarticipated in the development ot the AFFIRM Specification and Verification 
System, and served as the AFFIRM Project Leader in 1980-81. 


For further information about this and other Software Technology Seminars please check 
the appropriate box on the enclosed Reader Response Form or c¢ the Seminar Manager at 
Software Research Associates. 


Note: This and other SRA seminars can be presented “in-house” to larger groups of 
attendees at suDstantial overall savings and, in most cases, Dartially tallored to a client's 
specific needs. Please write for a copy of the SRA Software Technology Seminar Brochure. 


Soitware Research Associates 
PO. Box 24352 
San Francisco, CA 94126 


Phone: (415) 957-[441 — Telex: 340-255 
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SOFTWARE ENGINEERING AUTOMATED TOOLS INDEX 


As part of its continuing research activity in the Automated Software Engineering Tools 
area, Software Research Associates has assembled a comprehensive index of detailed data 
about a wice variety of software engineering support tools. 


Available March 1982, this Software Engineering Automated Tools Index will provide 
detailed information on approximately 500 different software engineering tools. 


Tools described in the Software Engineering Automated Tools Index fail into these major 
categories: 


o Software Requirements/Specification Tools 

o Software Design Tools 

o Software [Implementation (Programming) Tools 
o Software Quality Assurance Tools 

o Software Maintenance Tools 

o Software Project Management Tools 

o Cross-Environinent Tools 

o Miscellaneous Utility Systems 


The Index also includes a comprehensive By-Name Index, a By-Category Index, and a 
complete By-Supplier Index. Available information about obtaining each soltware system is 
also included. 


The information in the Software Engineering Automated Tools Index has deen gathered from 
a wide range of sources (Government, Industry, and Academia) over the past three years. 
Each automated tool is described in a singte "tool frame" that outlines such critical 
information as a tool's type and classification category, number of installations and orice, 
special features and exceptional characteristics, plus details about the needed execution 
environment. There are over 50 tool categories divided equally among the major system 
classes mentioned above. 


The Software Engineering Automated Tools Index is provided in convenient 3-ring Ddinder 
format, making it easy to survey the entire field of software engineering support tools, or 
to focus on just one area. This format makes it easy to incorporate quarterly updates that 
will be available to current users of the Software Engineering Automated Tools Index. The 
Two-Volume Tools Index costs are: U.S.A./Canada - $185.00; Foreign - $225.00. Costs for 
the quarterly HCAs (available on a subscription basis) are: U.S.A./Canaqa - $85.00; 
coreign - $115.00. 


For more information, or to reserve your copy of the Index, please check the appropriate 
boxes on the enclosed Reader Response Form. 


Note: Machine processible versions of the Software Engineering Automated Tools Index are 
also available on special license arrangement. Please write SRA [or detauls. 


Software Research Associates 
P.O. Box 2432 
San Francisco, CA 94126 


Phone: (415) 957-1441 - Telex: 340-235 
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Software Engineering Technology Seminars Spring 1982 Series 


ADVANCED SOFTWARE VALIDATION TECHNIQUES 


Modern methods of software engineering require use of advanced methods to as~ 
sure the installed quality of complex and critical software systems. 


This seminar addresses major issues facing the Verification and Validation com- 
munity in such areas as Symbolic Evaluation Methods, Verification Methods, Mu- 
tation Analysis, Functional Testing, Data Flow Analysis, and Domain Testing. 


Besides describing how these advanced concepts can be used in various ways in 
Quality Management programs, this seminar provides researchers and appliers of 
these technologies with detailed information about the payoffs as well as the 
Limitations of each method. For example, should mtation analysis be done on 
"large" programs? Or, should automated test data generation methods be used in 
a COBOL oriented environment? 


Attendees will learn about state-of-the-art concepts, and will receive a 
comprehensive set of course notes and, in addition, a set of reprints from the 
current technical literature. 


OUTLINE: 
SYMBOLIC EXECUTION TECHNIQUES 


Introduction 

Components of a Symbolic Execution System 

Problems in Implementing Symbolic Execution 
Detection of Anomalous Contructs 

Generation of Test Data 

Validation of Program Assertions 

Correspondence Between Programs and Specifications 
Partition Analysis 

Reliability of Symbolic Execution 


ADVANCES IN VERIFICATION 


Definitions 

Verification by Case Analysis 
Inductive Assertions 

Proofs with Symbolic Evaluation 
Reasoning from the Structure of Data 
Practical Alternatives 


MUTATION ANALYSIS 


De finition 

Testing Computer Programs 

Mutant Operators 

Relation to Other Testing Methods 
Practical Experience 

Systems That Have Been Built 
Relationship to Error Seeding 


SURVEY CF PROMISING TECHUNIUOUES 
Functional Testing 
Data Flow Analysis 


Error Seeding 
Domain Testing Strategy 


Software Research Associates -l- San Francisco, California 
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Sofiware Engineering Technology Seninars Spring 1982 Series 


THE INSTRUCTOR 
TIMOTHY BUDD is Assistant Professor of Computer Science at the University of 
Arizona. Professor Budd's research interests have focused on software en- 
zineering, program testing and validation techniques, and high level language 
implementation issues. He was a member of the research team which developed 
the Program Mutation Testing method, and has authored several papers on this 
and other areas of program validation technology. 


Professor Budd has the Ph.D. degree in Computer Science from Yale University. 


For further information about this and other Software Technology 
Seminars please contact the Seminar Manager at Software Research 
Associates... 


(415) 957-1441 
Or write to... 
Software Research Associates 


Pe Os Po0x 2 
San Francisco, California 94126 


tv 
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Seminar Outline Software Engineering Technology Fall 198} 


SOFTWARE QUALITY ASSURANCE TECHNOLOGY 


Developing procedures for assuring that a software system has the best possi- 
ble chance to operate without encountering "bugs" or "errors" is an activity 
that has formed a major focus of software engineering technology for nearly a 
decade. The goal of producing error-free software reliably and efficiently has 

eluded the best theoretical workers, while procedures for systematically 
analyzing and testing software through static and dynamic analysis has gained 
in popularity. Recent developments in software quality assurance make it pos- 
sible to have a reasonable expectation that software meets minimum standards 
of testing. This seminar focuses on the concepts, tools and techniques, con- 
temporary results, and prognosis for software quality assurance technology. 
Besides providing an investigation of state-of-the-art methods of program 
structure analysis (structured testing), the seminar presents a variety of ma- 
terial that deals with many alternative phases of software quality analysis. 
Attention is given not only to the theoretical aspects of the subject but also 
to practical results that can likely be achieved by use of known methods. 


Attendees receive an extensive set of notes and a copy of the tutorial text 
Software Testing and Validation Techniques, by Edward Miller and William &. 
Howden. Attendees will gain an increased understanding of quality assurance 
processes and procedures and will lesrn techniques that can be applied immedi- 
ately to quality assurance problems. 


OUTLINE: 
INTRODUCTION AND OVERVIEW 


Introduction to Methodology 

History of Testing and QA 

Limits of Technology 

Overview of Methodology 

Theoretical Implications/Limitations 


MANAGEMENT ASPECTS 


Organizational Setup 
Psychological Issues 
Level of Independence 
Typical Results of QA 
Case Studies 

Toolset Description 
Guidelines and Limits 


CODE INSPECTION AND STATIC ANALYSIS 


Goal of Static Analysis 

Code Inspection Procedures 
Typical Code Inspection Rules 
Role of Static Analyzers 

Case Studies 


Software Research Associates -l- San Francisco, California 
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Seminar Outline Software Engineering Technology Fall 1981 


ST PLANNING PROCEDURES 


See 





Objectives of Test Planning 

Role of Coverage Measures 

Structure of Programs (Graph Theory) 
Pure-Structured Programs’ Test Plans 
Hierarchical Decomposition Methods 
Statistics and Inferences 


TEST DATA SELECTION METHODS 


Critical Values Identified 
Optimum Choice of Specific Values 
Theoretical Justifications 
Relation to Proof of Correctness 
Examples 

Guidelines 


COVERAGE ANALYSIS 


Need for Coverage Measures 

Cl Defined and Explained 

Ct Defined and Explained 

Sl Defined and Explained 
Analysis for Ci/Si Evaluation 
3asis in Graph Theory 


DOCUMENTATION AND RETESTING 


Need for Documentation 

Data to Keep 

Retesting (Regression Testing) 
Change Control System 

Test Documentation Tools 


CASE STUDIES 


Role of Interactive Test Support System 
Small Example: ADD 

Medium Example: RKLASS, LEXICAL 

Large Example: FORM 

Statistics and Reliability Issues 
Recommendations 


AGENDA FOR RESEARCHERS 


THE INSTRUCTOR 


EDWARD F. MILLER, JR., is Technical Director of Software Research Associates, 
San Francisco, California, a firm devoted to advanced computer technology and 
software applications. His interests include software engineering management, 
software testing technology, software maintenance technology, automated tool 
design and computer architecture. 


Software Research Associates -2- San Francisco, California 
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Seminar Outline Software Engineering Technology Fall 1981 


Dr. Miller was previously Director of the Software Technology Center, Science 
Applications, Inc., San Francisco, and Director of the Program Validation Pro- 
ject at General Research Corporation, Santa Barbara, California. He received 
a BSEE at Iowa State University in 1962, an M.S. in Applied Mathematics at the 
University of Colorado in 1964, and the Ph.D. at the University of Maryland in 
1968 where he was an Instructor from 1964 to 1968. 


Dr. Miller is a member of the IEEE Computer Society, the ACM, SIAM and several 
honorary societies. He currently serves on several technical committees and 
is an Associate Technical Editor of COMPUTER Magazine. 


For further information about this and other Software Technology 
Seminars please contact the Seminar Manager at Software Research 
Associates... 


(415) 957-1441 


or write €o0... 


Software Research Associates 
P. 0. Box 2432 
San Francisco, California 94126 
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AUTOMATED SOFTWARE ENGINEERING TOOLS 


The central issue of software engineering lies in the use of automated tools 
that serve the software engineer by amplifying his capabilities. The software 
life-cycle can be divided into five phases: Requirements Analysis, Design, 
Implementation (Programming), Testing (Quality Assurance), and Maintenance. 
Specialized tools for each area have been found effective in many applica- 
tions, even while extensive tool-building research and development continues. 


Contemporary software engineering tools are exemplified by commercially avail- 
able tools that capture nearly every essential technical concept in good tool 
environments. Ranging from single tools that perform one important function 
(like a source-language instrumentor system) to integrated sets of tools that 
consolidate a variety of closely related functions, continued software en- 
gineering experience dictates the use of good tools == and in some cases the 
replacement or upgrade of bad tools. 


This seminar introduces the concepts of automated tools and how they relate to 
the software engineering life cycle, based on a state-of-the-art survey of 
contemporary (commercially or publicly available) software engineering tools. 
Besides providing an in-depth survey of tools that apply in all five areas, 
attention is devoted to system production support tools that aid in management 
of software development projects. Attention is also given to estimating when 
certain conceptually important tools are expected to be introduced in the 
market place in the near future. 


Attendees receive an extensive set of notes and a copy of the tutorial text 
Automated Tools for Software Engineering, by Edward Miller. Attendees will 
ee Ey ° ° 

Zain increased appreciation for good ae tool design, an increased under- 
standing of how tools interact, and a good feel for the present state-of-the- 
art in automated tools. 





OUTLINE: 
PHILOSOPHY OF AUTCMATION 


Motivating Forces 

General Principles 

Overview of Software Engineering Phases 
Overview of Tool Role 


TOOLS FOR SPECIFICATION/ REQUIREMENTS 


Analysis Tools 

Synthesis Tools 

Manual Versus Automated Versus Automatable Methodologies 
Contemporary Specifications/Requirements Tools 


TOOLS FOR DESIGN 


Principles of Design 

Modes of Design Assistance 

Limitations of Design Assistance 

Contemporary Design/Implementation Tools 

Interaction Between Tools and the Operating Environment 
Recommendations for Purchase/Lease Decisions 


TOOLS FOR PROGRAM IMPLEMENTATION 
Principles of Programming 
Programming Procedures 


Debugging Concepts 
Contemporary Program Implementation Tools 


Software Research Associates -l- San Franeciseo, California 


108 





Software Engineering Technology Seminars Spring 1982 Series 


TOOLS FOR QUALITY ASSURANCE AND TESTING 


Principles of Program Testing 

Role of Tools in Program Testing 

Limitations of Tools Applicable During Testing 
Specific Examples of Testing Tools 
Recommendations for Purchase/Buy Decision 


TOOLS FOR PROGRAM MAINTENANCE 


Principles of Software Maintenance 

Limitations of Automation for Program Maintenance 
Specific Example of Maintenance Tools 
Recommendations for Purchase/Buy Decision 


THE INSTRUCTOR 


EDWARD F. MILLER, JR. is Technical Director of Software Research Associates, 
San Francisco, Cali ornia, a firm devoted to advanced computer technology and 
software applications. His interests include software engineering management, 
software testing technology, software maintenance technology, automated tool 
design and cemputer architecture. 


Dr. Miller was previously Director of the Software Technology Center, Science 
Applications, Inc., San Francisco, and Director of the Program Validation Pro- 
ject at General Research Corporation, Santa Barbara, California. He received 
a BSEE at Iowa State University in 1962, an M.S. in Applied Mathematics at the 
University of Colorado in 1964, and the Ph.D. at the University of Maryland in 
1968 where he was an Instructor from 1964 to 1968. 


Dr. Miller is a member of the IEEE Computer Society, the ACM, SIAM and several 
honorary societies. He currently serves on several technical committees and 
is Associate Technical Editor of COMPUTER Magazine. 


For further information about this and other Software Technology 
Seminars please contact the Seminar Manager at Software Research 
Associates... 


or write to... 
Software Research Associates 


P. 0. Sox 2432 
San Francisco, California 94126 
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USER INTERFACE DESIGN PSYCHOLOGY 


User Interface Design, as a topic in its own right, has recently become the 
focus of significant design efforts. As the price/performance curve of 
hardware continues to show a decrease by a factor of 100 each 10 years, in- 
creasing emphasis can (in fact, must) be put on supporting user interactions. 
As a result, there is increased recognition in the computer industry of the 
essential importance of terms like "ease-of-learning™ and “ease of use." 


This seminar covers the application of selected information from the psychology 
of learning and of vision and time perception to the design of user/computer 
interfaces. 


Detailed Case Studies of commercial systems will be presented. Video taped 
demonstrations of these and some experimental systems will provide an awareness 
and some evaluation of the multitude of interaction techniques, approaches and 
devices that are now available. 


OUTLINE: 
INTRODUCTION 


Evolution of User I/F Technology 
Anatomy of the Seminar 

User L/F Dimensions |. 
Information Processing Model 
Futuristic User I/F Demo 


LEARNING THEORIES 


Sequential/Parallel Acquisition 
Linguistie/Spatial Materials 
Physiological 3asis for Thinking Styles 


CASE STUDY 1 


Graphics Editor Workstation 

Structural Model Generation Application 
Tablet/Menu Interaction 
Goals/Constraints/Rationale 


HUMAN MEMORY CHARACTERISTICS 


Short-Term/Long-Term Memory 
Recall Versus Recognition 
Spatial/Linguistiec Coding 

Role of Information Organization 


VISUAL PERCEPTION OVERVIEW 
Light/Space/Color/Time Sensitivities 
Visual Organization 
Display Symbols 

CASE STUDY 2 
Graphics Editor Workstation 
Color Charts and Graphs 


Mouse/Menu Interaction 
Goals/Constraints/Rationale 


Software Research Associates -l- San Francisco, California 
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STRESS IN USER/COMPUTER INTERACTION 


Causes of Stress 
What Can Be Done to Reduce 
Examples in Computer Systems 


INTERACTIVITY AND THE PERCEPTION OF TIME TIME 
User's Time Versus the Wall Clock 
Two Inceraction Models 
Case Study of a Database Interaction 


CASE STUDY 3 AND 4 
Desktop Computer Line Editor Study 
Application S/W Study 
Operating System Interaction Demonstration 


TEXT EDITOR DEMONSTRATIONS 


Line/Character/Screen Orienced 
Keyboard/Mouse/Tablet Devices 
Ease-of~Learning Versus Ease-of-Use 
Command Invocation Methods 


FUTURE CONSIDERATIONS 


Spatial interfaces 
Voice Interfaces 
Major Issues in the Field 


THE INSTRUCTOR 


DR. Jack GRIMES received his Ph.D from Iowa State University in Elececrical En- 
Zineering and "and Computer Science, his M.S. in Psychology and is currencly a doc- 
coral student in Applied Cognitive Psychology at che University of Oregon. 
Since 1971, he has been employed at Tektronix, Inc., in 3eaverton, Oregon, 
where he is currently a manager of advanced developmenc for deskcop computers. 


Dr. Grimes’ research interests have recently focused on understanding the na- 
cure of user-compucer interaction from the user's perspective. Previously, he 
worked in the areas of computer archicecture, silicon cechnology and program- 
ming syscems. 


Dr. Grimes was a participant in the China Technology Exchange Program in 1979, 
gave presencacCions ac the Compucer Architecture Workshop sponsored by Nixdorf 
in 1976 in West Germany, and parcicipated in che 2nd USA-Japan Conference held 
in Tokyo in 1975. Dr. Grimes has previously given a shorter version of this 
seminar at SIGGRAPH ‘80 and '81, che Sixth West Coast Compucer Faire and inter- 
nally at Tektronix. 


Ee a eS Pep ato Se ee SS a ee a ot a Oe Oe 


For further information about this and other Software Technology Seminars 
please concacc the Seminar Manager at Software Research Associaces... 
(415) 957-1441 


Or write to... 
Sofceware Research Associates 
P.O. Box 243% 
San Francisco, California 94126 


Software Research Associaces -2- San Francisco, California 
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SOFTWARE MAINTENANCE TECENOLOGY 


Software maintenance can often require 50% to 802 of the overall costs associ- 
ated with a software system's life cycle. Most of the activities of software 
maintenance involve detailed recordkeeping, incremental change to the software 
system, and analysis of the impace of changes. 


Current technology for software maintenance is in its infancy. Technical 
methods for analysis of complex and sophisticated computer programs can migrate 
from the research and development arena into practice only if care is taken in 
choosing the "right" algorithms and che “appropriate” methods of controlling 
change. This seminar focuses on methods for handling software maintenance prob- 
lems that are highly analytical in nature, but which can have immediate practi- 
cal benefit. Besides investigating various aspects of the maintenance problem, 
the seminar »resents methods of measuring and managing a variety of software 
Maintenance scenarios. 


Attendees will receive a comprehensive annotated bibliography of current 
literature pertaining to software maintenance technology, an extensive set of 
notes (ineluding case studies of typical maintenance situations), and reprints 
from the current technical literature. 


OUTLINE 3 


LtNTRODUECTION AND OVERVIEW 


Tmportance of Maintenance 
Purposes cf Maintenance 
Principles of Maintenance 


PROBLEMS OF MAINTENANCE 


User Knowledge 

Frogrammer Effectiveness, Availability 
System Quality 

Machine Requirements 

Environment Reliability 


PROGRAMMING ISSUES 
Types of Changes and Related Probiems 
Maintenance Scenarios 


Review Procedures, Documentation Methods 
Development Practices to Ease Maintenance Problems 


METRICS AND TESTING DURING MAINTENANCE 


Maintenance Metrics 
Functional Testing 
Coverage Testing 
SOFTWARE SYSTEM MANAGEMENT TECHNOLOGY 


Configuration Control 

Tese Libraries 

Error/Change Tracking 
MAINTENANCE AIDS AND TOOLS 


Software Tools 
Methodologies 


Software Research Associates -l- San Francisco, Caiifornia 
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MANAGEMENT ISSUES 


Scheduling for Maintenance 
Programmer Motivation 
Manpower Management 


SUMMARY AND RECOMMENDATIONS 


Overall Maintenance Plans 
Researchers’ Agenda 
Bibliography 


THE INSTRUCTOR 


EDWARD F. MILLER, JR., is Technical Director of Software Research Associates, 
San Francisco, California, a firm devoted to advanced computer technology and 
software applications. His interests include software engineering management, 
software testing technology, software maintenance technology, automated tool 
design and computer architecture. 


Dr. Miller was previously Director of the Software Technology Center, Science 
Applications, Inc., San Francisco, and Director of the Program Validation Pro- 
ject at General Research Corporation, Santa Barbara, Califernia. Se received a 
3SEE at Iowa State University in 1962, an M.S. in Applied Mathematics at the 
University of Colorado in 1964, and the Ph.D. at the University of Maryland in 
1968 where he was an Instructor from 1964 to 1968. 


Dr. Miller is a member of the IEEE Computer Society, the ACM, SIAM and several 
honorary societies. He currently serves on numerous technical committees and 
is Technical editor of COMPUTER Magazine. 
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For further information about this and other Software Technology 
Seminars please contact the Seminar Manager at Software Research 
Associates... 


(415) 957-1441 
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Software Research Associates 


NAME... 


Interactive Test Bed (ITB) for SRTRAN 
PURPOSE... 


Basic supoort of software quality assurance through systematic testing, bv 
assisting the user in achieving high values of Cl coverage. Assistance is 
provided by allowing the user to alter global data and analyzing the 


coverage of subsequent executions. Capahility to process Standard 
SRTRAN programs. 


SYNOPSIS... 


Basic capability for analy ing coverage results of executions’ in an 
interactive fashion. Also pvrovided is ability to alter data to program so as 
to alter program flow. 


Version current’v available only for Data General AOS environment. 


DESCRIPTION... 


A free- tanding ore-rocessor and testing aid for interactive analysis of 
coverage and execution results of SRTRAN programs and subprograms. 


The system consists of a SRTRAN instrumentor, a preprocessor which 
analyzes the data space of the program, and an interactive program which 


is linked to the specified test object. The preprocessor automatically 
generates subroutines which are used by the testbed specifically for the 
given test object. 


Coverage and execution results are reported when the user asks for that 
information. 


SPECIAL FEATURES... 


The ITB svstem automatically generates the code it needs to successfully 
test the test object. There exist macros which allows the user to set un 
an ITB in a few instructions. 


A trace feature is included which allows the user to follow execution of 
the test object ina segment by segment trace. This may de turned on or 
off at wilL 


Commands entered interactively are automatically stored away so as to 
give the user a complete record of his session on disk. Also availaDle is 
the ability to use this ‘hosting’ of previous sessions to be the input file 
to another test bed session. 


The entire data space can be saved at any time during a test ded session 
for the user to re-use later in the same session. 
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DOCUMENTATION... 
ITB comes with a Reference Manual 


SRA provides substantial related documentation on Software Quality 
Assurance and Software Maintenance. 


AVAILABILITY... 


The ITB system is currently onlv implemented on a Data General AOS 
environment. 


REQUIREMENTS... 
The system requires the nresence of a FORTRAN compiler and an 
SRTRAN preprocessor. 
CONTACT... 
Mr. Thomas E. Mapp 
Member of Technical Staff 
Software Research Associates 
P. O. Box 2432 
San Francisco, CA 94126 USA 


(415) 957-1441 


Updated: March 1981 
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NAME... 
COBOL Test Coverage Analysis Tool (TCAT/COBOL) 
PURPOSE... 


TCAT provides basic support of Software Quality Assurance through 
systematic testing by measuring the Cl and Pl coverage values for series of 


tests (Cl is the percentage of logical segments exercised and P1 is the 
percentage of paragraphs exercised). 


SYNOPSIS... 


TCAT provides a basic free-standing capability for automatic instrumentation 
of programs to analyze and report Ci and Pl coverage levels. TCAT 
processes ANS Standard COBOL programs, plus local machine dialect features 
depencing on the system version and host. 


Versions of TCAT are available for IBM, Univac, ACOS (Japan only), DEC 


VAX/VMS, Data General MV/8000, and ONYX C8002 (RM-COBOL/ Unix) 
computer environments. 


DESCRIPTION... 


TCAT is a free-standing pre-processor/post-processor system for batch 
oriented analysis of testing effectiveness of COBOL programs. 


The COBOL Test Coverage Analysis Tool consists of: (1) a comprehensive 
COBOL automatic instrumentor, INSTRU, (2) a set of run-time routines that 
are loaded and executed with the instrumented COBOL programs, called 
RUNTIME, and, (3) a standardized testing coverage analysis package called 
COVER. 


The pre-processing stage produces a Reference Listing, used to identify the 
logical segments and odaragraphs within the candidate COBOL program, and 
the post-execution stage of TCAT activity produces two forms of output: the 
Coverage Report and the Not Hit report. These show the percentage of 
coverage attainea by test(s) expressed in the Cl and Pl measures. In 
addition, the post-processing system generates a Histogram Report that shows 
the oroportion of times each segment and paragraph is executed. 


Coverage values attained by tests of the COBOL program are reported on a 
per-test, peftest-group, or an all-test cumulative Dasis. 


Coverage reporting nermally is defaulted to a predefined set of commonly 
used formats, out can be put completely under user control. 


SPECIAL FEATURES... 


The TCAT system can handle cumulative multi-run tests by storing standard 
coverage history records. Soecial Dlocking is used to reduce the size of the 
intermediate trace file. The level of system overneag with this method of 
intermediate file storage is reasonably low. 
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The €CAT system can handle multiple entry COBOL source modules as ‘veil 
as COBOL modules with multiple names. 


The Reference Listing produced by the pre-processor is specially annotated to 
show complete details of each logical segment in the program. The listing 
identifies the sense of each logical predicate outcome inthe COBOL logic, 
and provides statistics about the COBOL program that are useful for test 
module size comparisons and test difficulty estimation. 


Other features include run-time settable option settings. 


DOCUMENTATION... 


TCAT is supplied with a comprehensive Introduction and User’s Guide plus 
special installation support information as appropriate. 


Software Research Associates orovides substantial related documentation on 
Software Quality Assurance and Software Maintenance in the form of one-day 
and two-day Professional Development Seminars that can be made availaple 
for presentation upon request. 


AVAILABILITY Y... 


The COBOL TCAT system is available on a single-user dinary license 
agreement for a variety of computer systems (see above). 


Full documentation, installation-dependent information, and subdscription-type 
maintenance and upgrade service is also provided with the basic license 


agreement. Maintenance and upgrade service after the first year's use is also 
available. 


SYSTEM REQUIREMENTS... 


The TCAT system requires the presence of both a COBOL and a FORTRAN 
compiler. (The post-processing phase of TCAT is implemented in a pDortaple 

suDset of FORTRAN.) In addition, during execution of instrumented programs 
the TCAT system requires the use of one serial file. 


CONTACT... 
Christopher Walker 
Software Research Associates 
Pe Oe Box. 2432 
San Francisco, CA 94126 USA 


Phone: (415) 957-1441 —- Telex: 340-235 
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NAME... 
Extenued BASIC Validation Test Suite 
PURPOSE... 


Validation of BASIC interpreters/compilers which contain 
extensions similar to those found in the DEC BASIC-PLUS 
language. 


DESCRIPTION... 


The Extended BASIC Validation Test Suite is designed to 
validate the syntactic compatibility of a BASIC 
interpreter/compiler with the DEC BASIC-PLUS language . 


The test suite consists of over 200 test programs from the 
NBS Minimal BASIC Test Suite plus an additional 150 test 
programs which test the Extended BASIC language features 
of DEC BASIC-PLUS. The test programs cover standard 
capabilities, end cases, and exceptions for the language 
features. 


The extensions to the DEC BASIC-PLUS language include 
such features as matrix functions, block 1/0, control flow 
statements (WHILE, REPEAT, etc), string functions, and 
logical operators. All test groups are shown below. 


The output from the tests are fully machine processible, 
thereby facilitating later regression testing. 


Software Research Associates can offer either a complete 
testing service for a client's BASIC interpreter/compiler or 
the source code only for the Extended BASIC Test 
Programs. 


AVAILABILITY... 


The Extended BASIC Validation Test Suite is currently 
available for DEC BASIC-PLUS compatible implementations 
of BASIC. A future implementation will be compatibie with 
DG AOS/VS BASIC. SRA can also tailor a system to a 
client's specific language requirements. 


The DEC version of the Extended BASIC Test Suite is 


priced at $3200 for a single-user, single-site restricted 
source license. 


CONTACT... 
Mr. Mark Opperman 
Software Research Associates 
P.O. Box 2332 
San Franciseo, CA 94125 


(415) 957-1441 
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Group 


“1 MS 


CO wo ow 


Language Feature 


Simple Printing of string constants 


END and STOP 


PRINTing and simple assignment (LET) 


Extended BASIC Validation Test Suite Groups 


Control Statements and REM 


Variables 


Numerie Constants, Variables 


FOR-NENT 

Arrays 

Control Statements 
READ, DATA and RESTORE 


INPUT 


Imolementation-supplied Functions 


User-defined Functions 
Numeric Expressions 
Miscellaneous Checks 


Minimal BASIC Tests (Subtotal) 


Variables 

Arithmetie Operators 
Logical Operators 
String Operators 
Matrix Operators 


Mathematical Funetions 


Print Funetions 

String sunctions 
System Functions 
Matrix Funetions 


Input/Output Funetions 
Extended Statements 
Matrix Statements 
Statement Modifiers 
Block 1/O Statements 


Miseellaneous Features 
Immediate ‘lode 
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Extended BASIC Test Suite (Total) 
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SOFTWARE ENGINEERING AUTCMATED TOOLS INDEX PROSPECTUS 


The Software Engineering Automated Tools Index ("TOOLS INDEX") describes some 
600 automated tools that are available from commercial, governmental, indus- 
trial, and other sources in the United States and elsewhere in the world. All 
tools are categorized and cross-referenced in detail. 


1.0 CONTENTS 
Following is the structural contents of the TOOLS INDEX: 


Table of Contents 





1.0 Introduction 


1.1 Organization of TOOLS INDEX 
1.2 Contents of Tools Data Frames 
1.3 Cross-Reference Listings 

1.4 Updates and Corrections 

1.5 Sources of Information 


2.0 Tool Categories Listing 

3.0 Tool Name Cross-Reference Listing 

4.0 Tool Category Cross-Reference Listing 

5.0 Tool Supplier Cross-Reference Listing 

6.0 Supplier Address Listing 

7.0 SOFTWARE ENGINEERING AUTOMATED TOOLS INDEX DATA FRAMES (A-Z) 


8.0 References and 3ibliography 


2.0 AUTOMATED TOOL CATEGORIES 


The TOOLS INDEX is categorized based on each Tool's role in the software 
life cycle. The Tools are classified according to a scheme that provides 
a special "category number” for each major class of Tool. 


Following are the major categories used by the TOOLS INDEX (Reference at- 
tached detailed listing - “Automated Tool Categories"): 


- Requirement /Specification Tools 

- Software Design Tools 

- Software Implementation Tools 

- Software Testing Tools 

- Software Maintenance Tools 

- Software Project Management Tools 

- Language and Language Processing Systems 

- Utility Packages 

- Miscellaneous Support Tools 

- Research and Development Systems (Future Prototypes) 


Software Research Associates -1- San Francisco, California 
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SOFTWARE ENGINEERING AUTOMATED TOOLS INDEX PROSEECTUS 


3.0 AUTOMATED TOOL CROSS-REFERENCE LISTINGS 
The TOOLS INDEX provides a series of cross-reference listings to assist in 


locating specific tool data. 


3.1 Tool Name Listing 


Contains a three~field columnized description: 


Tool Name Category Number Supplier Name 


Listing is alphabetical by Tool Name. 


3.2 Tool Category Listing 


Contains a three~field columnized description: 


Category Number Tool Name Supplier Name 


Listing is in numeric sequence by Category Number. 


3.3 Tool Supplier Listing 


Contains a three-field columnized description: 
Supplier Name Tool Name Category Number 


Listing 18 alphabetical by Supplier Name. 


3.4 Tool Sunpvlier Address Listing 


Is an alphabetical listing, by Supplier Name, with addresses and 
telephone numbers. 


4.0 AUTOMATED TOOL DATA 
Tools are described on single "Frames’ and organized alphabetically dy 
Tool name. (Reference attached complete Frame, Figure 4.1, and actual 


sample, Figure 4-2.) 


The "Frame" contains a set of fields that describe various features of a 
Sarticiwlar 1001 ; 
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SOFTWARE ENGINEERING AUTOMATED TOOLS INDEX PROSPECTUS 


FIGURE 4-1: Conrcents of Automated Tool “Frame" 


Name 





Short name of tool (phrase describing tool use). 


Categorv 


Tool's numeric category (determined from "Automated Tools 
Categories" listing - assigned by SRA). 


Description 


Short (one paragraph) description of what the tool is and what the 
tool does. 


Number of Installations 

Number of Installations. 
Cost 

The cose for the system (including all options and variations). 
Configuration 

The configuration on which che cool operates. 
Contact 

Company name and mailing address to contact about chis tool. 
Telephone 

Telephone number of person to contact about this tool. 
Notes 


Special nores abouc the technical capbilities and features of this 
particular tool. 


Re ferences 
Any technical references that describe how this tool operates, its 
effectiveness, or its application (using standard bibliographic ci- 
tration format). 

Source 
The source of the informacion in the above (may be altered by SRA). 


Undated 


SRA date of latest revision/update of this Dlock of information. 
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FIGURE 4-2 


Sample Completed Tool "Frame" 


NAME... 

SRTRAN 1 (Baseline) 
CATEGORY... 

3.4 (Structured Programming Preprocessors) 
DESCRIPTION... 

Structured Programming Preprocessor for FORTRAN systems. 
NUMBER OF INSTALLATIONS... 

Approximately 15. 
COSTe me 

$750 for perpetual single-user binary license. 
CONFIGURATION... 


Portable to most FORTRAN environments. SRTRAN has been successfully in- 
stalled on I3M, Univac, Data General, DEC, and CDC computer systems. 


CONTACT... 
Software Research Associates 
P. O. Box 24 
San Francisco, CA 94126 

PHONE... 
(415) 957-1441 

NOLES.a. 
This is SRA’s own structured programming preprocessor. This "Daseline” 
system includes the standard set of Structured Programming constructs such 
as Te; 2. ELSE. .«2LSE IF..-eEND LF; CASE OF...CASE... ASE EL E..- END CASE, 
WHILE..END WHILE, REPEAT...END, etc. In addition, SRTRAN produces au~ 
tomatically indented, annotated listings of the source programs it 
processes. 
SRTRAN is documented in an extensive User's Manual. 

UPDATED sas 


ly Cetober 1931 


Software Research Associates 4 San Francisco, California 
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SOFTWARE ENGINEERING AUTOMATED TOOLS INDEX PROSPECTUS 


3.0 TOOLS INDEX UPDATES/CORRECTIONS 


6.0 


The TOOLS INDEX updates/corrections/deletions will be forwarded to sub- 
Scribers on a quarterly basis. SRA is continually modifying its computer- 
ized TOOLS INDEX files in order to reflect the most current information 
available. 


SUBSCRIPTION RATES 


The TOOLS INDEX, Volumes I & II, will be available January 1982. An Order 
Form is enclosed. Subscriptions for quarterly TOOLS INDEX updates will be 
available on a subscripton basis only at the rates quoted below. 





TOOLS INDEX QUARTERLY UPDATES 
2-Volume Set 
U.S.A. /Canada $185.00 U.S.A. /Canada $ 85.00/Yr. 
Foreign 3225-00 Foreign So007 yr. 


IMPORTANT INFORMATION 


U.S.A./Canada orders shipped 4th class book rate. Overseas mail 
shipped via sea mail (10-12 week delivery). 


For priority shipping to U.S.A./Canada, or airmail service (2 week 
delivery) to foreign countries, please add the following charges: 


Tools Index: U.S.A. /Canada $10.00/Set 
Foreign $50.00/Set 

Subscription U.S.A. /Canada $10.00/Order/Yr. 

Updates Foreign 925.00/Order/Yr. 


Tools Index price and quarterly subscription rates are subject 
to change without notice. 


Foreign checks must be in U.S. Dollars drawn on a U.S. dank. 


4.1 Computerized TOOLS INDEX 


Computer readable versions of the TOOLS INDEX are available on special re- 
quest. 
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For further information or ordering details, please contact: 


So 2 


Ms. Terryl Ostmo 

Software Research Associates 
P.O. Box 2432 

San Francisco, California 94126 


Telephone: (415) 957-1441 -- Telex: 340-235 
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